Sep 17 16:39:43.097772 ip-10-0-140-49 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 16:39:43.097785 ip-10-0-140-49 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 16:39:43.097795 ip-10-0-140-49 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 16:39:43.098010 ip-10-0-140-49 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 16:39:53.315337 ip-10-0-140-49 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 16:39:53.315353 ip-10-0-140-49 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 739ae247d5ab423a98c64bb501b18a9a --
Sep 17 16:41:59.099341 ip-10-0-140-49 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 16:42:02.613002 ip-10-0-140-49 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:42:02.613002 ip-10-0-140-49 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 16:42:02.613002 ip-10-0-140-49 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:42:02.613002 ip-10-0-140-49 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 16:42:02.613002 ip-10-0-140-49 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:42:02.613697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.613082    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 16:42:02.617364 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617344    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:42:02.617364 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617362    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:42:02.617364 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617366    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617370    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617373    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617376    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617380    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617383    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617386    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617389    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617392    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617395    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617397    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617400    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617404    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617407    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617409    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617412    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617415    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617418    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617421    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617423    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:42:02.617465 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617426    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617429    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617431    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617434    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617436    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617439    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617442    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617445    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617448    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617451    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617454    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617457    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617459    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617462    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617466    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617469    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617472    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617475    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617478    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617481    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:42:02.617985 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617484    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617488    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617492    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617496    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617499    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617502    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617505    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617508    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617511    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617514    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617517    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617520    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617523    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617526    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617528    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617531    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617535    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617538    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617541    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:42:02.618494 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617543    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617546    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617549    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617552    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617554    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617556    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617559    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617563    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617565    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617568    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617571    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617573    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617576    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617579    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617581    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617584    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617586    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617588    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617591    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617594    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:42:02.619028 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617597    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617599    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617602    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617605    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617607    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617610    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617613    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617615    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617618    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617620    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617624    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617629    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617632    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617635    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617638    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617641    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617644    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.617647    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618355    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618364    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:42:02.619549 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618368    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618371    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618374    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618378    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618380    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618383    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618386    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618389    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618391    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618394    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618398    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618400    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618403    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618406    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618408    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618411    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618413    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618416    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618419    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618421    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:42:02.620070 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618424    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618427    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618430    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618432    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618435    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618438    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618441    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618443    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618446    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618448    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618451    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618454    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618456    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618459    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618461    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618464    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618468    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618473    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618475    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:42:02.620569 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618478    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618481    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618484    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618486    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618489    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618491    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618494    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618497    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618500    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618502    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618505    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618508    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618511    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618514    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618517    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618520    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618522    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618525    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618527    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618530    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:42:02.621069 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618532    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618535    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618538    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618540    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618543    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618545    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618548    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618560    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618563    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618566    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618569    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618572    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618574    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618577    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618581    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618583    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618586    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618589    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618593    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618596    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:42:02.621590 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618599    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618602    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618605    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618608    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618611    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618613    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618616    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618622    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618626    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618629    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618632    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618635    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618637    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618640    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618643    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618646    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618651    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.618654    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619050    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:42:02.622101 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619055    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619058    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619061    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619064    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619067    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619070    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619074    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619077    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619080    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619083    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619086    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619089    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619092    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619095    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619098    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619101    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619104    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619107    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619110    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619113    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:42:02.622579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619116    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619119    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619122    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619125    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619128    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619131    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619133    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619136    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619139    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619144    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619147    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619149    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619152    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619154    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619157    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619160    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619163    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619165    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619168    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619170    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:42:02.623127 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619173    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619176    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619179    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619182    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619184    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619187    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619190    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619192    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619196    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619200    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619204    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619207    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619209    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619212    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619215    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619217    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619220    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619223    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619225    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:42:02.623642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619228    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619231    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619235    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619237    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619241    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619244    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619247    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619250    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619252    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619255    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619258    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619261    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619264    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619266    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619269    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619272    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619274    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619277    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619279    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:42:02.624146 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619282    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619284    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619289    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619292    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619295    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619298    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619301    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619303    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619306    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619308    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619311    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619314    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619316    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619319    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619321    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619325    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619328    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619331    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619333    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.619336    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:42:02.624642 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619409    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619442    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619448    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619452    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619456    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619459    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619464    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619468    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619471    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619474    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619478    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619481    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619484    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619487    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619492    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619495    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619498    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619501    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619506    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619509    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619512    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619515    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619519    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619523    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 16:42:02.625186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619526    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619529    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619532    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619536    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619540    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619543    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619546    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619549    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619553    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619556    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619559    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619563    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619566    2581 flags.go:64] FLAG: --enable-server="true"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619570    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619575    2581 flags.go:64] FLAG: --event-burst="100"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619578    2581 flags.go:64] FLAG: --event-qps="50"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619581    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619584    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619587    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619591    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619594    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619597    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619602    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619605    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619608    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 16:42:02.625780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619611    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619614    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619617    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619620    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619623    2581 flags.go:64] FLAG: --feature-gates=""
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619627    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619630    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619634    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619637    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619640    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619644    2581 flags.go:64] FLAG: --help="false"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619647    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-140-49.ec2.internal"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619650    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619653    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619657    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619660    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619663    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619666    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619669    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619672    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619675    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619678    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619682    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619685    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 17 16:42:02.626436 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619688    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619691    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619694    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619697    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619700    2581 flags.go:64] FLAG: --lock-file=""
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619704    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619707    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619710    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619715    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619718    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619721    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619723    2581 flags.go:64] FLAG: --logging-format="text"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619726    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619730    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619733    2581 flags.go:64] FLAG: --manifest-url=""
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619735    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619740    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619743    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619749    2581 flags.go:64] FLAG: --max-pods="110"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619752    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619755    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619758    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619761    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619764    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619767    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 16:42:02.627105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619770    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619777    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619780    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619783    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619786    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619789    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619795    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619798    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619801    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619804    2581 flags.go:64] FLAG: --port="10250"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619807    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619810    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07e140713c71ef826"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619813    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619817    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619820    2581 flags.go:64] FLAG: --register-node="true"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619823    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619827    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619830    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619832    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619835    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619839    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619842    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619845    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619848    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619851    2581 flags.go:64] FLAG: --runonce="false"
Sep 17 16:42:02.627842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619855    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619858    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619861    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619864    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619867    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619870    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619873    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619876    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619879    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619882    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619885    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619887    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619890    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619894    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619896    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619901    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619904    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619907    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619911    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619914    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619918    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619921    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619924    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619927    2581 flags.go:64] FLAG: --v="2"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619937    2581 flags.go:64] FLAG: --version="false"
Sep 17 16:42:02.628916 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619941    2581 flags.go:64] FLAG: --vmodule=""
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619944    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.619948    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620088    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620092    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620096    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620099    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620104    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620107    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620110    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620113    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620116    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620119    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620122    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620125    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620128    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620131    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620133    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620136    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620139    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:42:02.630064 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620141    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620145    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620148    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620151    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620154    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620156    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620159    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620163    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620166    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620168    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620171    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620174    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620176    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620179    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620181    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620184    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620187    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620190    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620194    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620197    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:42:02.630839 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620199    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620202    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620204    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620207    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620210    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620212    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620215    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620217    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620220    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620222    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620225    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620227    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620230    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620232    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620236    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620238    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620241    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620244    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620247    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:42:02.631579 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620249    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620253    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620257    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620260    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620263    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620266    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620269    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620271    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620274    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620277    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620279    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620282    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620285    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620287    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620290    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620292    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620295    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620298    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620300    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620303    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:42:02.632406 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620306    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620308    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620312    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620315    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620318    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620321    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620324    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620327    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620330    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620333    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620336    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620347    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620350    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620353    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620355    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620358    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620361    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620364    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620366    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:42:02.633067 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620369    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620371    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620374    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.620377    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.620882    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.629087    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.629107    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.629129    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.629139    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629236    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629244    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629249    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629255    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629260    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629267    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:42:02.633714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629272    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629276    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629281    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629285    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629290    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629294    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629298    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629303    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629307    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629311    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629316    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629320    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629324    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629329    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629333    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629337    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629351    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629357    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629362    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:42:02.634180 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629366    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629370    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629375    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629380    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629385    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629389    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629394    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629399    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629403    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629407    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629411    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629418    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629424    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629431    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629436    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629441    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629446    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629450    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629455    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629459    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:42:02.634714 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629464    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629468    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629473    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629479    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629483    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629487    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629492    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629496    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629501    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629506    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629511    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629515    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629519    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629523    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629527    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629531    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629536    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629541    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629545    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629550    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:42:02.635282 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629554    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629558    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629563    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629567    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629571    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629575    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629579    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629583    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629587    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629591    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629596    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629600    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629604    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629608    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629614    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629618    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629622    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629626    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629631    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629636    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:42:02.635790 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629640    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629644    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629648    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629653    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629657    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629661    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629666    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629670    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629675    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629679    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629683    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629688    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629692    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629696    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.629704    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:42:02.636331 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629890    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629899    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629903    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629908    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629912    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629917    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629922    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629926    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629930    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629934    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629938    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629942    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629947    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629951    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629956    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629981    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629987    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629991    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629995    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:42:02.636983 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.629999    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630003    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630007    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630011    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630015    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630019    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630025    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630029    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630033    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630037    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630042    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630046    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630051    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630055    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630059    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630063    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630066    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630071    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630075    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:42:02.637769 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630079    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630083    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630087    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630091    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630095    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630099    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630129    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630138    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630142    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630147    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630151    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630155    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630159    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630163    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630167    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630171    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630176    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630181    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630187    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630192    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:42:02.638683 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630196    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630201    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630208    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630214    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630219    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630224    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630228    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630232    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630237    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630241    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630246    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630250    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630255    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630259    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630263    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630267    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630272    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630276    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630280    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630284    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:42:02.643487 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630289    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630294    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630299    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630303    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630307    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630311    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630315    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630319    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630323    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630327    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630332    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630336    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630341    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630345    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630349    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630353    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630357    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630361    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630366    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:42:02.644128 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630369    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:42:02.644601 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:02.630374    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:42:02.644601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.630382    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:42:02.644601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.630542    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 16:42:02.644601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.633245    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 16:42:02.644601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.634333    2581 server.go:1019] "Starting client certificate rotation"
Sep 17 16:42:02.644601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.634438    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 16:42:02.644601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.634472    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 16:42:02.644601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.636862    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 16:42:02.644601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.638707    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 16:42:02.644601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.643286    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 17 16:42:02.644917 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.644887    2581 log.go:25] "Validated CRI v1 image API"
Sep 17 16:42:02.646429 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.646414    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 16:42:02.649381 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.649351    2581 fs.go:135] Filesystem UUIDs: map[26a99db9-6693-4ed5-92be-3e0e58e16bdf:/dev/nvme0n1p4 5de5ead7-1b2a-4179-9ec8-b8ea907eeebf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 17 16:42:02.649381 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.649372    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 16:42:02.653842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.653733    2581 manager.go:216] Machine: {Timestamp:2026-09-17 16:42:02.653265313 +0000 UTC m=+2.076203173 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102411 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21f9a46d289055beb99fd11cfe4b5f SystemUUID:ec21f9a4-6d28-9055-beb9-9fd11cfe4b5f BootID:739ae247-d5ab-423a-98c6-4bb501b18a9a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:d0:c0:79 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:d0:c0:79 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:4f:37:14:b9:e6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 16:42:02.653842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.653831    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 16:42:02.654025 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.653920    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 16:42:02.655533 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.655511    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 16:42:02.655679 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.655536    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-49.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 16:42:02.656722 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.656711    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 16:42:02.656761 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.656725    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 16:42:02.656761 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.656737    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 16:42:02.656761 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.656747    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 16:42:02.657937 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.657925    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 16:42:02.660796 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.660784    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 16:42:02.660896 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.660887    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 16:42:02.660932 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.660918    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 16:42:02.660932 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.660930    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 16:42:02.661017 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.660940    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 16:42:02.661017 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.660960    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 17 16:42:02.661017 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.660986    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 16:42:02.661433 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.661406    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 16:42:02.661433 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.661425    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 16:42:02.664720 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.664702    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 16:42:02.666928 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.666906    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 16:42:02.667026 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.666955    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 16:42:02.668096 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.668079    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 16:42:02.668183 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.668103    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 16:42:02.668183 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.668114    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 16:42:02.668183 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.668121    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 16:42:02.668183 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.668133    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 16:42:02.668183 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.668141    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 16:42:02.668183 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.668160    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 16:42:02.668183 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.668167    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 16:42:02.668183 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.668177    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 16:42:02.668183 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.668186    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 16:42:02.668479 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.668198    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 16:42:02.668479 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.668206    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 16:42:02.668479 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.668243    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 16:42:02.668479 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.668252    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 16:42:02.669367 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.669341    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 16:42:02.671780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.671762    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-49.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 16:42:02.672034 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.672020    2581 server.go:1268] "Started kubelet"
Sep 17 16:42:02.672220 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.672192    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 16:42:02.672305 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.672228    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 16:42:02.672305 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.672252    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-49.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 16:42:02.672305 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.672257    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 16:42:02.672305 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.672297    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 16:42:02.672762 ip-10-0-140-49 systemd[1]: Started Kubernetes Kubelet.
Sep 17 16:42:02.673382 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.673286    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 16:42:02.673587 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.673514    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 16:42:02.676512 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.676493    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 16:42:02.679032 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.679018    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 16:42:02.679090 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.679036    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 16:42:02.679309 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.679161    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-49.ec2.internal\" not found"
Sep 17 16:42:02.679559 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.679546    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 16:42:02.679634 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.679550    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 16:42:02.679634 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.679591    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 16:42:02.679732 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.679697    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 16:42:02.679732 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.679703    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 16:42:02.679897 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.679883    2581 factory.go:55] Registering systemd factory
Sep 17 16:42:02.679957 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.679901    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 17 16:42:02.680129 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.680117    2581 factory.go:153] Registering CRI-O factory
Sep 17 16:42:02.680177 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.680130    2581 factory.go:223] Registration of the crio container factory successfully
Sep 17 16:42:02.680207 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.680176    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 16:42:02.680207 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.680190    2581 factory.go:103] Registering Raw factory
Sep 17 16:42:02.680207 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.680201    2581 manager.go:1196] Started watching for new ooms in manager
Sep 17 16:42:02.680649 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.680639    2581 manager.go:349] Starting recovery of all containers
Sep 17 16:42:02.683470 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.683445    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-49.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 16:42:02.684150 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.684125    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 16:42:02.702690 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.702533    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7ghnd"
Sep 17 16:42:02.702690 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.702687    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7ghnd"
Sep 17 16:42:02.704332 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.704310    2581 manager.go:354] Recovery completed
Sep 17 16:42:02.709360 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.709344    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:42:02.710988 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.710958    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:42:02.711090 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.710998    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:42:02.711090 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.711007    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:42:02.711477 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.711461    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 16:42:02.711477 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.711475    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 16:42:02.711605 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.711490    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 16:42:02.711605 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.711475    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:42:02.712261 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.712246    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:42:02.712313 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.712276    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:42:02.712313 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.712286    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:42:02.712606 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.712594    2581 policy_none.go:49] "None policy: Start"
Sep 17 16:42:02.712635 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.712610    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 16:42:02.712635 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.712620    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 16:42:02.713572 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.713562    2581 policy_none.go:47] "Start"
Sep 17 16:42:02.754584 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.754571    2581 manager.go:337] "Starting Device Plugin manager"
Sep 17 16:42:02.767580 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.754602    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 16:42:02.767580 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.754611    2581 server.go:87] "Starting device plugin registration server"
Sep 17 16:42:02.767580 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.754805    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 16:42:02.767580 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.754822    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 16:42:02.767580 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.754887    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 16:42:02.767580 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.755003    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 16:42:02.767580 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.755015    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 16:42:02.767580 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.755483    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 16:42:02.767580 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.755527    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-49.ec2.internal\" not found"
Sep 17 16:42:02.828008 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.827975    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 16:42:02.829150 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.829134    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 16:42:02.829214 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.829185    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 16:42:02.829214 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.829207    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 16:42:02.829314 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.829249    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 16:42:02.831208 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.831186    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:42:02.855808 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.855782    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:42:02.859241 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.859226    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:42:02.859325 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.859256    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:42:02.859325 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.859281    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:42:02.859325 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.859308    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-49.ec2.internal"
Sep 17 16:42:02.866605 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.866561    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-49.ec2.internal"
Sep 17 16:42:02.866605 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.866581    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-49.ec2.internal\": node \"ip-10-0-140-49.ec2.internal\" not found"
Sep 17 16:42:02.876908 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.876886    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-49.ec2.internal\" not found"
Sep 17 16:42:02.930302 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.930281    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-49.ec2.internal"]
Sep 17 16:42:02.930382 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.930349    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:42:02.931661 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.931645    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:42:02.931732 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.931673    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:42:02.931732 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.931683    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:42:02.932872 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.932860    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:42:02.932937 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.932923    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:02.933003 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.932950    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:42:02.933533 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.933511    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:42:02.933622 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.933538    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:42:02.933622 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.933547    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:42:02.933622 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.933514    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:42:02.933622 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.933617    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:42:02.933776 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.933631    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:42:02.934712 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.934692    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:02.934788 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.934721    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:42:02.935438 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.935420    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:42:02.935510 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.935448    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:42:02.935510 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:02.935458    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:42:02.967606 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.967587    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-49.ec2.internal\" not found" node="ip-10-0-140-49.ec2.internal"
Sep 17 16:42:02.971941 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.971927    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-49.ec2.internal\" not found" node="ip-10-0-140-49.ec2.internal"
Sep 17 16:42:02.977992 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:02.977977    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-49.ec2.internal\" not found"
Sep 17 16:42:03.078902 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:03.078879    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-49.ec2.internal\" not found"
Sep 17 16:42:03.081111 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.081096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/188828d9adc2fecd35744998d86d21ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal\" (UID: \"188828d9adc2fecd35744998d86d21ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:03.081176 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.081118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/188828d9adc2fecd35744998d86d21ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal\" (UID: \"188828d9adc2fecd35744998d86d21ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:03.081176 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.081141    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af9930b89e342a5bca676a2c71f00917-config\") pod \"kube-apiserver-proxy-ip-10-0-140-49.ec2.internal\" (UID: \"af9930b89e342a5bca676a2c71f00917\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:03.179583 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:03.179529    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-49.ec2.internal\" not found"
Sep 17 16:42:03.181726 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.181712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/188828d9adc2fecd35744998d86d21ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal\" (UID: \"188828d9adc2fecd35744998d86d21ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:03.181787 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.181735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/188828d9adc2fecd35744998d86d21ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal\" (UID: \"188828d9adc2fecd35744998d86d21ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:03.181787 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.181753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af9930b89e342a5bca676a2c71f00917-config\") pod \"kube-apiserver-proxy-ip-10-0-140-49.ec2.internal\" (UID: \"af9930b89e342a5bca676a2c71f00917\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:03.181849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.181801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af9930b89e342a5bca676a2c71f00917-config\") pod \"kube-apiserver-proxy-ip-10-0-140-49.ec2.internal\" (UID: \"af9930b89e342a5bca676a2c71f00917\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:03.181849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.181824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/188828d9adc2fecd35744998d86d21ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal\" (UID: \"188828d9adc2fecd35744998d86d21ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:03.181909 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.181825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/188828d9adc2fecd35744998d86d21ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal\" (UID: \"188828d9adc2fecd35744998d86d21ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:03.270903 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.270873    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:03.275883 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.275636    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:03.279631 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:03.279612    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-49.ec2.internal\" not found"
Sep 17 16:42:03.379995 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:03.379952    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-49.ec2.internal\" not found"
Sep 17 16:42:03.480482 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:03.480419    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-49.ec2.internal\" not found"
Sep 17 16:42:03.580912 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:03.580873    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-49.ec2.internal\" not found"
Sep 17 16:42:03.634174 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.634149    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 16:42:03.634535 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.634287    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 16:42:03.666685 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.666656    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:42:03.681532 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:03.681510    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-49.ec2.internal\" not found"
Sep 17 16:42:03.707940 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.707895    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 16:37:02 +0000 UTC" deadline="2028-05-20 07:24:37.679196379 +0000 UTC"
Sep 17 16:42:03.707940 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.707936    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14654h42m33.97126391s"
Sep 17 16:42:03.782452 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:03.782409    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-49.ec2.internal\" not found"
Sep 17 16:42:03.883098 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:03.883065    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-49.ec2.internal\" not found"
Sep 17 16:42:03.888075 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.888055    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:42:03.965551 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:03.965527    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188828d9adc2fecd35744998d86d21ff.slice/crio-2404b7f593ce0daa0ff79ada1e772d21f96a2935ea7f22970aca7bacb2b0e4cf WatchSource:0}: Error finding container 2404b7f593ce0daa0ff79ada1e772d21f96a2935ea7f22970aca7bacb2b0e4cf: Status 404 returned error can't find the container with id 2404b7f593ce0daa0ff79ada1e772d21f96a2935ea7f22970aca7bacb2b0e4cf
Sep 17 16:42:03.965990 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:03.965975    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9930b89e342a5bca676a2c71f00917.slice/crio-81dc9542b5c635fd75221f6b6704f58e713608283fece21080805c262ba064f2 WatchSource:0}: Error finding container 81dc9542b5c635fd75221f6b6704f58e713608283fece21080805c262ba064f2: Status 404 returned error can't find the container with id 81dc9542b5c635fd75221f6b6704f58e713608283fece21080805c262ba064f2
Sep 17 16:42:03.972239 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.972225    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:42:03.979980 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.979953    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:03.992254 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.992230    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 16:42:03.994105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:03.993368    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal"
Sep 17 16:42:04.004770 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.004749    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 16:42:04.082913 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.082895    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:42:04.661921 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.661889    2581 apiserver.go:52] "Watching apiserver"
Sep 17 16:42:04.669015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.668939    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 16:42:04.669382 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.669355    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-rkpps","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal","openshift-multus/multus-additional-cni-plugins-dzjw5","openshift-multus/multus-mpgnj","kube-system/kube-apiserver-proxy-ip-10-0-140-49.ec2.internal","openshift-image-registry/node-ca-kfplg","openshift-multus/network-metrics-daemon-24wdv","openshift-network-diagnostics/network-check-target-qz854","openshift-network-operator/iptables-alerter-sc2rc","openshift-ovn-kubernetes/ovnkube-node-tr5db","kube-system/global-pull-secret-syncer-lgk7x","kube-system/konnectivity-agent-vscfj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk","openshift-cluster-node-tuning-operator/tuned-2npqv"]
Sep 17 16:42:04.670943 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.670918    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:04.671062 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:04.671019    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:04.672502 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.672054    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vscfj"
Sep 17 16:42:04.673835 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.673343    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.676880 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.674357    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 16:42:04.676880 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.674626    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pbmvg\""
Sep 17 16:42:04.676880 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.674763    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 16:42:04.676880 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.674919    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.676880 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.675317    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 16:42:04.676880 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.675660    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rkpps"
Sep 17 16:42:04.676880 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.676145    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 16:42:04.676880 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.676181    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 16:42:04.676880 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.676149    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 16:42:04.676880 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.676393    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wffxt\""
Sep 17 16:42:04.676880 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.676399    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 16:42:04.677504 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.677314    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 16:42:04.677504 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.677456    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2pk8w\""
Sep 17 16:42:04.677653 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.677630    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 16:42:04.678889 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.677932    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 16:42:04.678889 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.678176    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5xd54\""
Sep 17 16:42:04.678889 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.678368    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kfplg"
Sep 17 16:42:04.680478 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.679271    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sc2rc"
Sep 17 16:42:04.680478 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.679328    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 16:42:04.682847 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.681052    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8g9nh\""
Sep 17 16:42:04.682847 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.681300    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 16:42:04.682847 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.681475    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 16:42:04.682847 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.681869    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 16:42:04.682847 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.682046    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:42:04.682847 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.682088    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.682847 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.682115    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 16:42:04.682847 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.682261    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jkk7q\""
Sep 17 16:42:04.682847 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.682335    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.682847 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.682363    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 16:42:04.688933 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.686706    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 16:42:04.688933 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.687109    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9k5ns\""
Sep 17 16:42:04.688933 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.687156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.688933 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.687354    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:42:04.688933 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.687762    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 16:42:04.688933 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.687843    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:04.688933 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:04.687923    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:04.688933 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.688058    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:04.688933 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:04.688133    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:04.688933 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.688273    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 16:42:04.688933 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.688481    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 16:42:04.688933 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.688692    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r2qhr\""
Sep 17 16:42:04.688933 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.688840    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-socket-dir\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.688933 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.688880    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-cnibin\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.689617 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c12bc685-04a3-484b-b606-3306605095e5-cni-binary-copy\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.689617 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-multus-conf-dir\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.689617 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21fa7670-4703-438a-9c6b-3caa76193e5b-agent-certs\") pod \"konnectivity-agent-vscfj\" (UID: \"21fa7670-4703-438a-9c6b-3caa76193e5b\") " pod="kube-system/konnectivity-agent-vscfj"
Sep 17 16:42:04.689617 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21fa7670-4703-438a-9c6b-3caa76193e5b-konnectivity-ca\") pod \"konnectivity-agent-vscfj\" (UID: \"21fa7670-4703-438a-9c6b-3caa76193e5b\") " pod="kube-system/konnectivity-agent-vscfj"
Sep 17 16:42:04.689617 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-system-cni-dir\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.689866 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-var-lib-cni-bin\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.689866 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689722    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 16:42:04.689866 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0aa3902-ccf1-49cf-ad48-aafa74d040f0-tmp-dir\") pod \"node-resolver-rkpps\" (UID: \"d0aa3902-ccf1-49cf-ad48-aafa74d040f0\") " pod="openshift-dns/node-resolver-rkpps"
Sep 17 16:42:04.689866 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689776    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vqnx5\""
Sep 17 16:42:04.689866 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-registration-dir\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.689866 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689815    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-system-cni-dir\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.689866 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-cni-binary-copy\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.690210 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.690210 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689907    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 16:42:04.690210 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-hostroot\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.690210 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.689942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-run-multus-certs\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.690210 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 16:42:04.690210 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-etc-selinux\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.690210 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-sys-fs\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.690210 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs\") pod \"network-metrics-daemon-24wdv\" (UID: \"d306fc98-5114-4b68-a4f7-bfaaf0e419d1\") " pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:04.690583 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.690583 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-multus-cni-dir\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.690583 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h69bt\" (UniqueName: \"kubernetes.io/projected/d0aa3902-ccf1-49cf-ad48-aafa74d040f0-kube-api-access-h69bt\") pod \"node-resolver-rkpps\" (UID: \"d0aa3902-ccf1-49cf-ad48-aafa74d040f0\") " pod="openshift-dns/node-resolver-rkpps"
Sep 17 16:42:04.690583 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndsn2\" (UniqueName: \"kubernetes.io/projected/965e6eb0-6a53-451c-9331-4c48bc36d1cb-kube-api-access-ndsn2\") pod \"iptables-alerter-sc2rc\" (UID: \"965e6eb0-6a53-451c-9331-4c48bc36d1cb\") " pod="openshift-network-operator/iptables-alerter-sc2rc"
Sep 17 16:42:04.690583 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc6zj\" (UniqueName: \"kubernetes.io/projected/d5dc52de-ad02-4a17-ab94-629161ff514b-kube-api-access-hc6zj\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.690583 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-cnibin\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.690583 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c12bc685-04a3-484b-b606-3306605095e5-multus-daemon-config\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.690583 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690524    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 16:42:04.690956 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0513c985-c365-41ee-959d-b02794d96aab-host\") pod \"node-ca-kfplg\" (UID: \"0513c985-c365-41ee-959d-b02794d96aab\") " pod="openshift-image-registry/node-ca-kfplg"
Sep 17 16:42:04.690956 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/965e6eb0-6a53-451c-9331-4c48bc36d1cb-host-slash\") pod \"iptables-alerter-sc2rc\" (UID: \"965e6eb0-6a53-451c-9331-4c48bc36d1cb\") " pod="openshift-network-operator/iptables-alerter-sc2rc"
Sep 17 16:42:04.690956 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.690956 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjdqp\" (UniqueName: \"kubernetes.io/projected/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-kube-api-access-hjdqp\") pod \"network-metrics-daemon-24wdv\" (UID: \"d306fc98-5114-4b68-a4f7-bfaaf0e419d1\") " pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:04.690956 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.690956 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78krv\" (UniqueName: \"kubernetes.io/projected/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-kube-api-access-78krv\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.691269 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.690997    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-multus-socket-dir-parent\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.691269 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.691198    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 16:42:04.691269 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.691201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-run-netns\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.691269 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.691245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-var-lib-cni-multus\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.691454 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.691275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-var-lib-kubelet\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.691454 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.691299    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmvcs\" (UniqueName: \"kubernetes.io/projected/0513c985-c365-41ee-959d-b02794d96aab-kube-api-access-tmvcs\") pod \"node-ca-kfplg\" (UID: \"0513c985-c365-41ee-959d-b02794d96aab\") " pod="openshift-image-registry/node-ca-kfplg"
Sep 17 16:42:04.691454 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.691321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-etc-kubernetes\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.691454 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.691337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtk84\" (UniqueName: \"kubernetes.io/projected/c12bc685-04a3-484b-b606-3306605095e5-kube-api-access-qtk84\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.691454 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.691432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0513c985-c365-41ee-959d-b02794d96aab-serviceca\") pod \"node-ca-kfplg\" (UID: \"0513c985-c365-41ee-959d-b02794d96aab\") " pod="openshift-image-registry/node-ca-kfplg"
Sep 17 16:42:04.691454 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.691450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/965e6eb0-6a53-451c-9331-4c48bc36d1cb-iptables-alerter-script\") pod \"iptables-alerter-sc2rc\" (UID: \"965e6eb0-6a53-451c-9331-4c48bc36d1cb\") " pod="openshift-network-operator/iptables-alerter-sc2rc"
Sep 17 16:42:04.691781 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.691468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-device-dir\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.691781 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.691482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-run-k8s-cni-cncf-io\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.691781 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.691501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0aa3902-ccf1-49cf-ad48-aafa74d040f0-hosts-file\") pod \"node-resolver-rkpps\" (UID: \"d0aa3902-ccf1-49cf-ad48-aafa74d040f0\") " pod="openshift-dns/node-resolver-rkpps"
Sep 17 16:42:04.691923 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.691815    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 16:42:04.695679 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.695656    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 16:42:04.717701 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.717682    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-49ftw"
Sep 17 16:42:04.727191 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.726867    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-49ftw"
Sep 17 16:42:04.781258 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.781241    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 16:42:04.792088 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-etc-kubernetes\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.792197 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0513c985-c365-41ee-959d-b02794d96aab-serviceca\") pod \"node-ca-kfplg\" (UID: \"0513c985-c365-41ee-959d-b02794d96aab\") " pod="openshift-image-registry/node-ca-kfplg"
Sep 17 16:42:04.792197 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-device-dir\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.792197 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-kubelet\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.792197 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-cni-bin\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.792390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-run-k8s-cni-cncf-io\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.792390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0aa3902-ccf1-49cf-ad48-aafa74d040f0-hosts-file\") pod \"node-resolver-rkpps\" (UID: \"d0aa3902-ccf1-49cf-ad48-aafa74d040f0\") " pod="openshift-dns/node-resolver-rkpps"
Sep 17 16:42:04.792390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-socket-dir\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.792390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9800468e-129a-4126-9483-59b5fe270d5f-etc-tuned\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.792390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-run-systemd\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.792390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-run-ovn-kubernetes\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.792390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-cni-netd\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.792390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4158398-2987-4869-9185-3d8962e8ee54-ovnkube-config\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.792390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-cnibin\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.792390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c12bc685-04a3-484b-b606-3306605095e5-cni-binary-copy\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792403    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21fa7670-4703-438a-9c6b-3caa76193e5b-agent-certs\") pod \"konnectivity-agent-vscfj\" (UID: \"21fa7670-4703-438a-9c6b-3caa76193e5b\") " pod="kube-system/konnectivity-agent-vscfj"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-run-netns\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4158398-2987-4869-9185-3d8962e8ee54-env-overrides\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-systemd\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9800468e-129a-4126-9483-59b5fe270d5f-tmp\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-system-cni-dir\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-cni-binary-copy\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-run-multus-certs\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-modprobe-d\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xvtx\" (UniqueName: \"kubernetes.io/projected/9800468e-129a-4126-9483-59b5fe270d5f-kube-api-access-7xvtx\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-run-ovn\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-log-socket\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-sysctl-conf\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.792848 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792706    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0513c985-c365-41ee-959d-b02794d96aab-serviceca\") pod \"node-ca-kfplg\" (UID: \"0513c985-c365-41ee-959d-b02794d96aab\") " pod="openshift-image-registry/node-ca-kfplg"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h69bt\" (UniqueName: \"kubernetes.io/projected/d0aa3902-ccf1-49cf-ad48-aafa74d040f0-kube-api-access-h69bt\") pod \"node-resolver-rkpps\" (UID: \"d0aa3902-ccf1-49cf-ad48-aafa74d040f0\") " pod="openshift-dns/node-resolver-rkpps"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6zj\" (UniqueName: \"kubernetes.io/projected/d5dc52de-ad02-4a17-ab94-629161ff514b-kube-api-access-hc6zj\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-sysconfig\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792840    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-etc-openvswitch\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-run-openvswitch\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c12bc685-04a3-484b-b606-3306605095e5-multus-daemon-config\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/965e6eb0-6a53-451c-9331-4c48bc36d1cb-host-slash\") pod \"iptables-alerter-sc2rc\" (UID: \"965e6eb0-6a53-451c-9331-4c48bc36d1cb\") " pod="openshift-network-operator/iptables-alerter-sc2rc"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792959    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-device-dir\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.792991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-slash\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78krv\" (UniqueName: \"kubernetes.io/projected/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-kube-api-access-78krv\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793036    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-run-k8s-cni-cncf-io\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-multus-socket-dir-parent\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0aa3902-ccf1-49cf-ad48-aafa74d040f0-hosts-file\") pod \"node-resolver-rkpps\" (UID: \"d0aa3902-ccf1-49cf-ad48-aafa74d040f0\") " pod="openshift-dns/node-resolver-rkpps"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-run-netns\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-var-lib-kubelet\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.793697 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-run-netns\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-kubernetes\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-run\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793178    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-etc-kubernetes\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-var-lib-openvswitch\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-socket-dir\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-cnibin\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793647    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793909    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c12bc685-04a3-484b-b606-3306605095e5-cni-binary-copy\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-run-multus-certs\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.793982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/965e6eb0-6a53-451c-9331-4c48bc36d1cb-host-slash\") pod \"iptables-alerter-sc2rc\" (UID: \"965e6eb0-6a53-451c-9331-4c48bc36d1cb\") " pod="openshift-network-operator/iptables-alerter-sc2rc"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794159    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-system-cni-dir\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-var-lib-kubelet\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4158398-2987-4869-9185-3d8962e8ee54-ovn-node-metrics-cert\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtk84\" (UniqueName: \"kubernetes.io/projected/c12bc685-04a3-484b-b606-3306605095e5-kube-api-access-qtk84\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/965e6eb0-6a53-451c-9331-4c48bc36d1cb-iptables-alerter-script\") pod \"iptables-alerter-sc2rc\" (UID: \"965e6eb0-6a53-451c-9331-4c48bc36d1cb\") " pod="openshift-network-operator/iptables-alerter-sc2rc"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-multus-conf-dir\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21fa7670-4703-438a-9c6b-3caa76193e5b-konnectivity-ca\") pod \"konnectivity-agent-vscfj\" (UID: \"21fa7670-4703-438a-9c6b-3caa76193e5b\") " pod="kube-system/konnectivity-agent-vscfj"
Sep 17 16:42:04.794601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-var-lib-kubelet\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794603    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-multus-socket-dir-parent\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-cni-binary-copy\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-systemd-units\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4158398-2987-4869-9185-3d8962e8ee54-ovnkube-script-lib\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794723    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-system-cni-dir\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-multus-conf-dir\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c12bc685-04a3-484b-b606-3306605095e5-multus-daemon-config\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.794979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-var-lib-cni-bin\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0aa3902-ccf1-49cf-ad48-aafa74d040f0-tmp-dir\") pod \"node-resolver-rkpps\" (UID: \"d0aa3902-ccf1-49cf-ad48-aafa74d040f0\") " pod="openshift-dns/node-resolver-rkpps"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-var-lib-cni-bin\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-registration-dir\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-hostroot\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21fa7670-4703-438a-9c6b-3caa76193e5b-konnectivity-ca\") pod \"konnectivity-agent-vscfj\" (UID: \"21fa7670-4703-438a-9c6b-3caa76193e5b\") " pod="kube-system/konnectivity-agent-vscfj"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795183    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-system-cni-dir\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-etc-selinux\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.795451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795264    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/965e6eb0-6a53-451c-9331-4c48bc36d1cb-iptables-alerter-script\") pod \"iptables-alerter-sc2rc\" (UID: \"965e6eb0-6a53-451c-9331-4c48bc36d1cb\") " pod="openshift-network-operator/iptables-alerter-sc2rc"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795282    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-registration-dir\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-etc-selinux\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-sys-fs\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-hostroot\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tljk5\" (UniqueName: \"kubernetes.io/projected/c4158398-2987-4869-9185-3d8962e8ee54-kube-api-access-tljk5\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-sys-fs\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0aa3902-ccf1-49cf-ad48-aafa74d040f0-tmp-dir\") pod \"node-resolver-rkpps\" (UID: \"d0aa3902-ccf1-49cf-ad48-aafa74d040f0\") " pod="openshift-dns/node-resolver-rkpps"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs\") pod \"network-metrics-daemon-24wdv\" (UID: \"d306fc98-5114-4b68-a4f7-bfaaf0e419d1\") " pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:04.795563    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-multus-cni-dir\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndsn2\" (UniqueName: \"kubernetes.io/projected/965e6eb0-6a53-451c-9331-4c48bc36d1cb-kube-api-access-ndsn2\") pod \"iptables-alerter-sc2rc\" (UID: \"965e6eb0-6a53-451c-9331-4c48bc36d1cb\") " pod="openshift-network-operator/iptables-alerter-sc2rc"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:04.795657    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs podName:d306fc98-5114-4b68-a4f7-bfaaf0e419d1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:05.295636318 +0000 UTC m=+4.718574165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs") pod "network-metrics-daemon-24wdv" (UID: "d306fc98-5114-4b68-a4f7-bfaaf0e419d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-multus-cni-dir\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmc9z\" (UniqueName: \"kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z\") pod \"network-check-target-qz854\" (UID: \"469c498c-7671-4d9b-ae5d-ee75d043b4d4\") " pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-sys\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-node-log\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.796247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-cnibin\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0513c985-c365-41ee-959d-b02794d96aab-host\") pod \"node-ca-kfplg\" (UID: \"0513c985-c365-41ee-959d-b02794d96aab\") " pod="openshift-image-registry/node-ca-kfplg"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.795960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-cnibin\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.796007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.796033    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0513c985-c365-41ee-959d-b02794d96aab-host\") pod \"node-ca-kfplg\" (UID: \"0513c985-c365-41ee-959d-b02794d96aab\") " pod="openshift-image-registry/node-ca-kfplg"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.796051    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/667acd31-9d7e-4010-bbb3-f18b17232f48-kubelet-config\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.796076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-sysctl-d\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.796088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5dc52de-ad02-4a17-ab94-629161ff514b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.796098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-host\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.796124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjdqp\" (UniqueName: \"kubernetes.io/projected/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-kube-api-access-hjdqp\") pod \"network-metrics-daemon-24wdv\" (UID: \"d306fc98-5114-4b68-a4f7-bfaaf0e419d1\") " pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.796153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.796176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-var-lib-cni-multus\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.796200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmvcs\" (UniqueName: \"kubernetes.io/projected/0513c985-c365-41ee-959d-b02794d96aab-kube-api-access-tmvcs\") pod \"node-ca-kfplg\" (UID: \"0513c985-c365-41ee-959d-b02794d96aab\") " pod="openshift-image-registry/node-ca-kfplg"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.796222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/667acd31-9d7e-4010-bbb3-f18b17232f48-dbus\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.796242    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-lib-modules\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.796334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c12bc685-04a3-484b-b606-3306605095e5-host-var-lib-cni-multus\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.797067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.796624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.799003 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.798982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21fa7670-4703-438a-9c6b-3caa76193e5b-agent-certs\") pod \"konnectivity-agent-vscfj\" (UID: \"21fa7670-4703-438a-9c6b-3caa76193e5b\") " pod="kube-system/konnectivity-agent-vscfj"
Sep 17 16:42:04.801550 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.801528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.810752 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.810700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndsn2\" (UniqueName: \"kubernetes.io/projected/965e6eb0-6a53-451c-9331-4c48bc36d1cb-kube-api-access-ndsn2\") pod \"iptables-alerter-sc2rc\" (UID: \"965e6eb0-6a53-451c-9331-4c48bc36d1cb\") " pod="openshift-network-operator/iptables-alerter-sc2rc"
Sep 17 16:42:04.810842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.810784    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmvcs\" (UniqueName: \"kubernetes.io/projected/0513c985-c365-41ee-959d-b02794d96aab-kube-api-access-tmvcs\") pod \"node-ca-kfplg\" (UID: \"0513c985-c365-41ee-959d-b02794d96aab\") " pod="openshift-image-registry/node-ca-kfplg"
Sep 17 16:42:04.811143 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.811119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78krv\" (UniqueName: \"kubernetes.io/projected/d04dd819-1558-44a9-9fe9-cdb6fd48ec82-kube-api-access-78krv\") pod \"multus-additional-cni-plugins-dzjw5\" (UID: \"d04dd819-1558-44a9-9fe9-cdb6fd48ec82\") " pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:04.811228 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.811168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtk84\" (UniqueName: \"kubernetes.io/projected/c12bc685-04a3-484b-b606-3306605095e5-kube-api-access-qtk84\") pod \"multus-mpgnj\" (UID: \"c12bc685-04a3-484b-b606-3306605095e5\") " pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:04.812245 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.812203    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc6zj\" (UniqueName: \"kubernetes.io/projected/d5dc52de-ad02-4a17-ab94-629161ff514b-kube-api-access-hc6zj\") pod \"aws-ebs-csi-driver-node-ft2gk\" (UID: \"d5dc52de-ad02-4a17-ab94-629161ff514b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:04.812763 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.812741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h69bt\" (UniqueName: \"kubernetes.io/projected/d0aa3902-ccf1-49cf-ad48-aafa74d040f0-kube-api-access-h69bt\") pod \"node-resolver-rkpps\" (UID: \"d0aa3902-ccf1-49cf-ad48-aafa74d040f0\") " pod="openshift-dns/node-resolver-rkpps"
Sep 17 16:42:04.814274 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.814253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjdqp\" (UniqueName: \"kubernetes.io/projected/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-kube-api-access-hjdqp\") pod \"network-metrics-daemon-24wdv\" (UID: \"d306fc98-5114-4b68-a4f7-bfaaf0e419d1\") " pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:04.835490 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.835420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal" event={"ID":"188828d9adc2fecd35744998d86d21ff","Type":"ContainerStarted","Data":"2404b7f593ce0daa0ff79ada1e772d21f96a2935ea7f22970aca7bacb2b0e4cf"}
Sep 17 16:42:04.836698 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.836675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-49.ec2.internal" event={"ID":"af9930b89e342a5bca676a2c71f00917","Type":"ContainerStarted","Data":"81dc9542b5c635fd75221f6b6704f58e713608283fece21080805c262ba064f2"}
Sep 17 16:42:04.897293 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-kubernetes\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.897390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-run\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.897390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-var-lib-openvswitch\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.897390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4158398-2987-4869-9185-3d8962e8ee54-ovn-node-metrics-cert\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.897390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-var-lib-kubelet\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.897579 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-systemd-units\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.897579 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4158398-2987-4869-9185-3d8962e8ee54-ovnkube-script-lib\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.897579 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tljk5\" (UniqueName: \"kubernetes.io/projected/c4158398-2987-4869-9185-3d8962e8ee54-kube-api-access-tljk5\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.897579 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmc9z\" (UniqueName: \"kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z\") pod \"network-check-target-qz854\" (UID: \"469c498c-7671-4d9b-ae5d-ee75d043b4d4\") " pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:04.897579 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-sys\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.897579 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-node-log\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.897579 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/667acd31-9d7e-4010-bbb3-f18b17232f48-kubelet-config\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:04.897579 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-systemd-units\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.897579 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-sysctl-d\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-host\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/667acd31-9d7e-4010-bbb3-f18b17232f48-dbus\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-lib-modules\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-sysctl-d\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-kubelet\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-cni-bin\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9800468e-129a-4126-9483-59b5fe270d5f-etc-tuned\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-run-systemd\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-run-ovn-kubernetes\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-cni-netd\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897829    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/667acd31-9d7e-4010-bbb3-f18b17232f48-dbus\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4158398-2987-4869-9185-3d8962e8ee54-ovnkube-config\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-run-netns\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4158398-2987-4869-9185-3d8962e8ee54-env-overrides\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-systemd\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9800468e-129a-4126-9483-59b5fe270d5f-tmp\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.898015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-modprobe-d\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-lib-modules\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.897993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvtx\" (UniqueName: \"kubernetes.io/projected/9800468e-129a-4126-9483-59b5fe270d5f-kube-api-access-7xvtx\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-kubelet\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-run-ovn\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-log-socket\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-sysctl-conf\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-sysconfig\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-etc-openvswitch\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-var-lib-kubelet\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-etc-openvswitch\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4158398-2987-4869-9185-3d8962e8ee54-ovnkube-script-lib\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-run-openvswitch\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-cni-bin\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-host\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-slash\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-slash\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.898863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-run-ovn\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-run-systemd\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4158398-2987-4869-9185-3d8962e8ee54-ovnkube-config\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-run-ovn-kubernetes\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-cni-netd\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:04.898764    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898799    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/667acd31-9d7e-4010-bbb3-f18b17232f48-kubelet-config\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898800    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-node-log\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:04.898814    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret podName:667acd31-9d7e-4010-bbb3-f18b17232f48 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:05.398798357 +0000 UTC m=+4.821736200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret") pod "global-pull-secret-syncer-lgk7x" (UID: "667acd31-9d7e-4010-bbb3-f18b17232f48") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898838    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-sys\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898851    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-log-socket\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-sysctl-conf\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-sysconfig\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898874    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-systemd\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-run\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-run-openvswitch\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.899873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-modprobe-d\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.900803 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9800468e-129a-4126-9483-59b5fe270d5f-etc-kubernetes\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.900803 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.898959    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-var-lib-openvswitch\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.900803 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.899001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4158398-2987-4869-9185-3d8962e8ee54-host-run-netns\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.900803 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.899287    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4158398-2987-4869-9185-3d8962e8ee54-env-overrides\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.901360 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.901338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4158398-2987-4869-9185-3d8962e8ee54-ovn-node-metrics-cert\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.902010 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.901948    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9800468e-129a-4126-9483-59b5fe270d5f-etc-tuned\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.902776 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.902738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9800468e-129a-4126-9483-59b5fe270d5f-tmp\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.909767 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.909745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tljk5\" (UniqueName: \"kubernetes.io/projected/c4158398-2987-4869-9185-3d8962e8ee54-kube-api-access-tljk5\") pod \"ovnkube-node-tr5db\" (UID: \"c4158398-2987-4869-9185-3d8962e8ee54\") " pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:04.910752 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:04.910732    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:04.910845 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:04.910758    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:04.910845 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:04.910770    2581 projected.go:196] Error preparing data for projected volume kube-api-access-kmc9z for pod openshift-network-diagnostics/network-check-target-qz854: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:04.910845 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:04.910820    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z podName:469c498c-7671-4d9b-ae5d-ee75d043b4d4 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:05.410804954 +0000 UTC m=+4.833742808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kmc9z" (UniqueName: "kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z") pod "network-check-target-qz854" (UID: "469c498c-7671-4d9b-ae5d-ee75d043b4d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:04.912409 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.912367    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xvtx\" (UniqueName: \"kubernetes.io/projected/9800468e-129a-4126-9483-59b5fe270d5f-kube-api-access-7xvtx\") pod \"tuned-2npqv\" (UID: \"9800468e-129a-4126-9483-59b5fe270d5f\") " pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:04.987897 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:04.987868    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vscfj"
Sep 17 16:42:04.999878 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:04.999843    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21fa7670_4703_438a_9c6b_3caa76193e5b.slice/crio-f821810584923ada4353c6051e7f51e8cf6f26d0abab3408e05843089a2efe4b WatchSource:0}: Error finding container f821810584923ada4353c6051e7f51e8cf6f26d0abab3408e05843089a2efe4b: Status 404 returned error can't find the container with id f821810584923ada4353c6051e7f51e8cf6f26d0abab3408e05843089a2efe4b
Sep 17 16:42:05.000540 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.000401    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dzjw5"
Sep 17 16:42:05.007759 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:05.007734    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04dd819_1558_44a9_9fe9_cdb6fd48ec82.slice/crio-2bb7dc5595c830517861a3167a293ea0b6c58e13a3be0e16c3e7cce9bf2b582f WatchSource:0}: Error finding container 2bb7dc5595c830517861a3167a293ea0b6c58e13a3be0e16c3e7cce9bf2b582f: Status 404 returned error can't find the container with id 2bb7dc5595c830517861a3167a293ea0b6c58e13a3be0e16c3e7cce9bf2b582f
Sep 17 16:42:05.008915 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.008893    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mpgnj"
Sep 17 16:42:05.017995 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.017956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rkpps"
Sep 17 16:42:05.025566 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.025549    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kfplg"
Sep 17 16:42:05.034099 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.034081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sc2rc"
Sep 17 16:42:05.043772 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.043752    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk"
Sep 17 16:42:05.051380 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.051364    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2npqv"
Sep 17 16:42:05.057888 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.057872    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:05.225295 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.225221    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:42:05.301319 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.301290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs\") pod \"network-metrics-daemon-24wdv\" (UID: \"d306fc98-5114-4b68-a4f7-bfaaf0e419d1\") " pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:05.301462 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:05.301449    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:05.301532 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:05.301517    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs podName:d306fc98-5114-4b68-a4f7-bfaaf0e419d1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:06.301498471 +0000 UTC m=+5.724436329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs") pod "network-metrics-daemon-24wdv" (UID: "d306fc98-5114-4b68-a4f7-bfaaf0e419d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:05.402692 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.402649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:05.402865 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:05.402824    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:05.402929 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:05.402899    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret podName:667acd31-9d7e-4010-bbb3-f18b17232f48 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:06.402878277 +0000 UTC m=+5.825816121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret") pod "global-pull-secret-syncer-lgk7x" (UID: "667acd31-9d7e-4010-bbb3-f18b17232f48") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:05.503201 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.503130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmc9z\" (UniqueName: \"kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z\") pod \"network-check-target-qz854\" (UID: \"469c498c-7671-4d9b-ae5d-ee75d043b4d4\") " pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:05.503326 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:05.503284    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:05.503326 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:05.503300    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:05.503326 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:05.503309    2581 projected.go:196] Error preparing data for projected volume kube-api-access-kmc9z for pod openshift-network-diagnostics/network-check-target-qz854: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:05.503455 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:05.503364    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z podName:469c498c-7671-4d9b-ae5d-ee75d043b4d4 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:06.503345733 +0000 UTC m=+5.926283578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kmc9z" (UniqueName: "kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z") pod "network-check-target-qz854" (UID: "469c498c-7671-4d9b-ae5d-ee75d043b4d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:05.729053 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.729018    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 16:37:04 +0000 UTC" deadline="2028-04-27 22:01:29.171225484 +0000 UTC"
Sep 17 16:42:05.729053 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.729045    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14117h19m23.442183649s"
Sep 17 16:42:05.838713 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.838649    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzjw5" event={"ID":"d04dd819-1558-44a9-9fe9-cdb6fd48ec82","Type":"ContainerStarted","Data":"2bb7dc5595c830517861a3167a293ea0b6c58e13a3be0e16c3e7cce9bf2b582f"}
Sep 17 16:42:05.839837 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:05.839806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vscfj" event={"ID":"21fa7670-4703-438a-9c6b-3caa76193e5b","Type":"ContainerStarted","Data":"f821810584923ada4353c6051e7f51e8cf6f26d0abab3408e05843089a2efe4b"}
Sep 17 16:42:05.969077 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:05.969053    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12bc685_04a3_484b_b606_3306605095e5.slice/crio-c06fc5e9973209a15e818f2c302b27ba9d8382f690cabb4036c57ef0ae2ad53e WatchSource:0}: Error finding container c06fc5e9973209a15e818f2c302b27ba9d8382f690cabb4036c57ef0ae2ad53e: Status 404 returned error can't find the container with id c06fc5e9973209a15e818f2c302b27ba9d8382f690cabb4036c57ef0ae2ad53e
Sep 17 16:42:05.970759 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:05.970673    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dc52de_ad02_4a17_ab94_629161ff514b.slice/crio-41582777850d0422510d22c4502846ca9a622da977712a44b5fc2b3e487fa5ca WatchSource:0}: Error finding container 41582777850d0422510d22c4502846ca9a622da977712a44b5fc2b3e487fa5ca: Status 404 returned error can't find the container with id 41582777850d0422510d22c4502846ca9a622da977712a44b5fc2b3e487fa5ca
Sep 17 16:42:05.973203 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:05.973124    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0513c985_c365_41ee_959d_b02794d96aab.slice/crio-377b621aa4bca422d99f947f8c2626d9981f7aa1a7f9fa3d0c691075380b9f34 WatchSource:0}: Error finding container 377b621aa4bca422d99f947f8c2626d9981f7aa1a7f9fa3d0c691075380b9f34: Status 404 returned error can't find the container with id 377b621aa4bca422d99f947f8c2626d9981f7aa1a7f9fa3d0c691075380b9f34
Sep 17 16:42:05.974233 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:05.974181    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9800468e_129a_4126_9483_59b5fe270d5f.slice/crio-5c3ad2383f9a1226e670d7cbf43c4de888609abaa2db4d6c1efdbd48621ce774 WatchSource:0}: Error finding container 5c3ad2383f9a1226e670d7cbf43c4de888609abaa2db4d6c1efdbd48621ce774: Status 404 returned error can't find the container with id 5c3ad2383f9a1226e670d7cbf43c4de888609abaa2db4d6c1efdbd48621ce774
Sep 17 16:42:05.975211 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:05.975186    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965e6eb0_6a53_451c_9331_4c48bc36d1cb.slice/crio-ed37922042766b9644bda2f1d8eec847447e6dfb6fc61ebc854a6609f37fa895 WatchSource:0}: Error finding container ed37922042766b9644bda2f1d8eec847447e6dfb6fc61ebc854a6609f37fa895: Status 404 returned error can't find the container with id ed37922042766b9644bda2f1d8eec847447e6dfb6fc61ebc854a6609f37fa895
Sep 17 16:42:05.976469 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:05.976445    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0aa3902_ccf1_49cf_ad48_aafa74d040f0.slice/crio-3282aaf862b15ee944ec5a96c7274f78b37bb5b382b870d5e4475d6eab36ca2b WatchSource:0}: Error finding container 3282aaf862b15ee944ec5a96c7274f78b37bb5b382b870d5e4475d6eab36ca2b: Status 404 returned error can't find the container with id 3282aaf862b15ee944ec5a96c7274f78b37bb5b382b870d5e4475d6eab36ca2b
Sep 17 16:42:05.977118 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:05.977035    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4158398_2987_4869_9185_3d8962e8ee54.slice/crio-be36ae4b2c62defdc2dbdd8423d44ad4b7bc299685a39785ef4b6cd6deff0994 WatchSource:0}: Error finding container be36ae4b2c62defdc2dbdd8423d44ad4b7bc299685a39785ef4b6cd6deff0994: Status 404 returned error can't find the container with id be36ae4b2c62defdc2dbdd8423d44ad4b7bc299685a39785ef4b6cd6deff0994
Sep 17 16:42:06.310520 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.310332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs\") pod \"network-metrics-daemon-24wdv\" (UID: \"d306fc98-5114-4b68-a4f7-bfaaf0e419d1\") " pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:06.310654 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:06.310561    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:06.310654 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:06.310635    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs podName:d306fc98-5114-4b68-a4f7-bfaaf0e419d1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:08.310615264 +0000 UTC m=+7.733553129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs") pod "network-metrics-daemon-24wdv" (UID: "d306fc98-5114-4b68-a4f7-bfaaf0e419d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:06.411871 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.411836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:06.412051 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:06.412008    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:06.412130 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:06.412068    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret podName:667acd31-9d7e-4010-bbb3-f18b17232f48 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:08.412049656 +0000 UTC m=+7.834987516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret") pod "global-pull-secret-syncer-lgk7x" (UID: "667acd31-9d7e-4010-bbb3-f18b17232f48") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:06.512902 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.512819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmc9z\" (UniqueName: \"kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z\") pod \"network-check-target-qz854\" (UID: \"469c498c-7671-4d9b-ae5d-ee75d043b4d4\") " pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:06.513070 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:06.513038    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:06.513070 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:06.513059    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:06.513070 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:06.513071    2581 projected.go:196] Error preparing data for projected volume kube-api-access-kmc9z for pod openshift-network-diagnostics/network-check-target-qz854: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:06.513233 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:06.513129    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z podName:469c498c-7671-4d9b-ae5d-ee75d043b4d4 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:08.513111078 +0000 UTC m=+7.936048934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kmc9z" (UniqueName: "kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z") pod "network-check-target-qz854" (UID: "469c498c-7671-4d9b-ae5d-ee75d043b4d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:06.729419 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.729289    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 16:37:04 +0000 UTC" deadline="2028-03-12 12:46:26.063556799 +0000 UTC"
Sep 17 16:42:06.729419 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.729414    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13004h4m19.334147319s"
Sep 17 16:42:06.832603 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.832535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:06.832746 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:06.832658    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:06.832811 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.832746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:06.832863 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:06.832825    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:06.832918 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.832881    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:06.833000 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:06.832958    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:06.855466 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.855435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sc2rc" event={"ID":"965e6eb0-6a53-451c-9331-4c48bc36d1cb","Type":"ContainerStarted","Data":"ed37922042766b9644bda2f1d8eec847447e6dfb6fc61ebc854a6609f37fa895"}
Sep 17 16:42:06.858888 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.858860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2npqv" event={"ID":"9800468e-129a-4126-9483-59b5fe270d5f","Type":"ContainerStarted","Data":"5c3ad2383f9a1226e670d7cbf43c4de888609abaa2db4d6c1efdbd48621ce774"}
Sep 17 16:42:06.864017 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.862931    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk" event={"ID":"d5dc52de-ad02-4a17-ab94-629161ff514b","Type":"ContainerStarted","Data":"41582777850d0422510d22c4502846ca9a622da977712a44b5fc2b3e487fa5ca"}
Sep 17 16:42:06.865542 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.865518    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mpgnj" event={"ID":"c12bc685-04a3-484b-b606-3306605095e5","Type":"ContainerStarted","Data":"c06fc5e9973209a15e818f2c302b27ba9d8382f690cabb4036c57ef0ae2ad53e"}
Sep 17 16:42:06.876414 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.875708    2581 generic.go:356] "Generic (PLEG): container finished" podID="188828d9adc2fecd35744998d86d21ff" containerID="eddc0a458ea44bd1e9beebf11c9ad4e9cbe390dd1f7284986f87ff7641de58b4" exitCode=0
Sep 17 16:42:06.876414 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.875779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal" event={"ID":"188828d9adc2fecd35744998d86d21ff","Type":"ContainerDied","Data":"eddc0a458ea44bd1e9beebf11c9ad4e9cbe390dd1f7284986f87ff7641de58b4"}
Sep 17 16:42:06.884865 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.884817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-49.ec2.internal" event={"ID":"af9930b89e342a5bca676a2c71f00917","Type":"ContainerStarted","Data":"1d7e9922b3ea0a46e626f16c8a14d892ac0355255942900db8ba7fee7c12aaae"}
Sep 17 16:42:06.892054 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.891930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db" event={"ID":"c4158398-2987-4869-9185-3d8962e8ee54","Type":"ContainerStarted","Data":"be36ae4b2c62defdc2dbdd8423d44ad4b7bc299685a39785ef4b6cd6deff0994"}
Sep 17 16:42:06.908081 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.908059    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rkpps" event={"ID":"d0aa3902-ccf1-49cf-ad48-aafa74d040f0","Type":"ContainerStarted","Data":"3282aaf862b15ee944ec5a96c7274f78b37bb5b382b870d5e4475d6eab36ca2b"}
Sep 17 16:42:06.912814 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:06.912787    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kfplg" event={"ID":"0513c985-c365-41ee-959d-b02794d96aab","Type":"ContainerStarted","Data":"377b621aa4bca422d99f947f8c2626d9981f7aa1a7f9fa3d0c691075380b9f34"}
Sep 17 16:42:07.933199 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:07.933162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal" event={"ID":"188828d9adc2fecd35744998d86d21ff","Type":"ContainerStarted","Data":"26a31206af9b6ee51ae2b81ca7a066213dcc40b47f20bb956623205721915618"}
Sep 17 16:42:07.953825 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:07.953771    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-49.ec2.internal" podStartSLOduration=4.953753244 podStartE2EDuration="4.953753244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:06.912580823 +0000 UTC m=+6.335518687" watchObservedRunningTime="2026-09-17 16:42:07.953753244 +0000 UTC m=+7.376691124"
Sep 17 16:42:08.330748 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:08.330507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs\") pod \"network-metrics-daemon-24wdv\" (UID: \"d306fc98-5114-4b68-a4f7-bfaaf0e419d1\") " pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:08.340205 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:08.331351    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:08.340205 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:08.331441    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs podName:d306fc98-5114-4b68-a4f7-bfaaf0e419d1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:12.331419705 +0000 UTC m=+11.754357603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs") pod "network-metrics-daemon-24wdv" (UID: "d306fc98-5114-4b68-a4f7-bfaaf0e419d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:08.431327 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:08.431268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:08.431490 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:08.431402    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:08.431490 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:08.431460    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret podName:667acd31-9d7e-4010-bbb3-f18b17232f48 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:12.431444253 +0000 UTC m=+11.854382117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret") pod "global-pull-secret-syncer-lgk7x" (UID: "667acd31-9d7e-4010-bbb3-f18b17232f48") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:08.531928 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:08.531872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmc9z\" (UniqueName: \"kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z\") pod \"network-check-target-qz854\" (UID: \"469c498c-7671-4d9b-ae5d-ee75d043b4d4\") " pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:08.532101 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:08.532059    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:08.532101 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:08.532077    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:08.532101 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:08.532089    2581 projected.go:196] Error preparing data for projected volume kube-api-access-kmc9z for pod openshift-network-diagnostics/network-check-target-qz854: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:08.532253 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:08.532144    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z podName:469c498c-7671-4d9b-ae5d-ee75d043b4d4 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:12.532125892 +0000 UTC m=+11.955063757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kmc9z" (UniqueName: "kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z") pod "network-check-target-qz854" (UID: "469c498c-7671-4d9b-ae5d-ee75d043b4d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:08.831819 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:08.831780    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:08.832001 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:08.831897    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:08.832344 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:08.832319    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:08.832434 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:08.832418    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:08.832489 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:08.832477    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:08.832622 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:08.832554    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:10.829936 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:10.829904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:10.830378 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:10.830046    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:10.830378 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:10.830234    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:10.830378 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:10.830361    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:10.830560 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:10.830386    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:10.830560 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:10.830487    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:12.364649 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:12.364607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs\") pod \"network-metrics-daemon-24wdv\" (UID: \"d306fc98-5114-4b68-a4f7-bfaaf0e419d1\") " pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:12.365167 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:12.364781    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:12.365167 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:12.364857    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs podName:d306fc98-5114-4b68-a4f7-bfaaf0e419d1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:20.36483736 +0000 UTC m=+19.787775206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs") pod "network-metrics-daemon-24wdv" (UID: "d306fc98-5114-4b68-a4f7-bfaaf0e419d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:12.466024 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:12.465512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:12.466024 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:12.465656    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:12.466024 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:12.465714    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret podName:667acd31-9d7e-4010-bbb3-f18b17232f48 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:20.465696391 +0000 UTC m=+19.888634237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret") pod "global-pull-secret-syncer-lgk7x" (UID: "667acd31-9d7e-4010-bbb3-f18b17232f48") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:12.567446 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:12.566818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmc9z\" (UniqueName: \"kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z\") pod \"network-check-target-qz854\" (UID: \"469c498c-7671-4d9b-ae5d-ee75d043b4d4\") " pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:12.567446 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:12.567027    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:12.567446 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:12.567048    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:12.567446 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:12.567061    2581 projected.go:196] Error preparing data for projected volume kube-api-access-kmc9z for pod openshift-network-diagnostics/network-check-target-qz854: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:12.567446 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:12.567118    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z podName:469c498c-7671-4d9b-ae5d-ee75d043b4d4 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:20.567100153 +0000 UTC m=+19.990037998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kmc9z" (UniqueName: "kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z") pod "network-check-target-qz854" (UID: "469c498c-7671-4d9b-ae5d-ee75d043b4d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:12.830383 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:12.830305    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:12.830538 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:12.830384    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:12.830766 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:12.830745    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:12.830867 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:12.830843    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:12.830928 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:12.830901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:12.831044 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:12.830998    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:14.830172 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:14.830141    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:14.830172 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:14.830164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:14.830727 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:14.830141    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:14.830727 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:14.830261    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:14.830727 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:14.830363    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:14.830727 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:14.830500    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:16.830336 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:16.830298    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:16.830781 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:16.830298    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:16.830781 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:16.830465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:16.830781 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:16.830429    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:16.830781 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:16.830544    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:16.830781 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:16.830679    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:18.829461 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:18.829431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:18.829818 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:18.829469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:18.829818 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:18.829433    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:18.829818 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:18.829554    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:18.829818 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:18.829668    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:18.829818 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:18.829750    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:20.426459 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:20.426427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs\") pod \"network-metrics-daemon-24wdv\" (UID: \"d306fc98-5114-4b68-a4f7-bfaaf0e419d1\") " pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:20.426843 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:20.426580    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:20.426843 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:20.426636    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs podName:d306fc98-5114-4b68-a4f7-bfaaf0e419d1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:36.426620448 +0000 UTC m=+35.849558302 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs") pod "network-metrics-daemon-24wdv" (UID: "d306fc98-5114-4b68-a4f7-bfaaf0e419d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:20.527727 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:20.527697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:20.527891 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:20.527837    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:20.527993 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:20.527895    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret podName:667acd31-9d7e-4010-bbb3-f18b17232f48 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:36.527879872 +0000 UTC m=+35.950817730 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret") pod "global-pull-secret-syncer-lgk7x" (UID: "667acd31-9d7e-4010-bbb3-f18b17232f48") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:20.628596 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:20.628562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmc9z\" (UniqueName: \"kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z\") pod \"network-check-target-qz854\" (UID: \"469c498c-7671-4d9b-ae5d-ee75d043b4d4\") " pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:20.628756 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:20.628739    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:20.628832 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:20.628767    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:20.628832 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:20.628780    2581 projected.go:196] Error preparing data for projected volume kube-api-access-kmc9z for pod openshift-network-diagnostics/network-check-target-qz854: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:20.628929 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:20.628852    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z podName:469c498c-7671-4d9b-ae5d-ee75d043b4d4 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:36.628831355 +0000 UTC m=+36.051769213 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kmc9z" (UniqueName: "kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z") pod "network-check-target-qz854" (UID: "469c498c-7671-4d9b-ae5d-ee75d043b4d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:20.830003 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:20.829916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:20.830003 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:20.829921    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:20.830217 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:20.829927    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:20.830217 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:20.830119    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:20.830324 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:20.830212    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:20.830373 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:20.830334    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:22.830856 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:22.830827    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:22.831147 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:22.830903    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:22.831147 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:22.830991    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:22.831147 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:22.831011    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:22.831147 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:22.831095    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:22.831293 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:22.831176    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:23.966139 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.965746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2npqv" event={"ID":"9800468e-129a-4126-9483-59b5fe270d5f","Type":"ContainerStarted","Data":"2329267265e7e5cc12a99a57245edf0e3db23bda62225612188fdb61a9d71ed9"}
Sep 17 16:42:23.967329 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.967306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk" event={"ID":"d5dc52de-ad02-4a17-ab94-629161ff514b","Type":"ContainerStarted","Data":"3fdec0c13d287161d23e4bb48ec96e855317f8416d279b0bf594c6197ef4438c"}
Sep 17 16:42:23.968745 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.968723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mpgnj" event={"ID":"c12bc685-04a3-484b-b606-3306605095e5","Type":"ContainerStarted","Data":"aca6a01d9bc9061eb533b09cfe0e0860fb45795491e195efcacf4d9193088d9d"}
Sep 17 16:42:23.971432 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.971417    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 16:42:23.971768 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.971751    2581 generic.go:356] "Generic (PLEG): container finished" podID="c4158398-2987-4869-9185-3d8962e8ee54" containerID="500a1c03f3f58abf89c6649dc1a416a5ad35f1be20b9508bbf27c9c6e10b1366" exitCode=1
Sep 17 16:42:23.971832 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.971811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db" event={"ID":"c4158398-2987-4869-9185-3d8962e8ee54","Type":"ContainerStarted","Data":"fb0e10777c9abf3e810df364550c8a17c83c708fa7ef4a23681454bc687773ea"}
Sep 17 16:42:23.971889 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.971873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db" event={"ID":"c4158398-2987-4869-9185-3d8962e8ee54","Type":"ContainerStarted","Data":"5bd230471a705b31a2321545c0ed63f32e4543c81342da2b4df9ca68bdc29f2f"}
Sep 17 16:42:23.971943 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.971911    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db" event={"ID":"c4158398-2987-4869-9185-3d8962e8ee54","Type":"ContainerStarted","Data":"b8781c81b8dd40cd608015a693e1a5173b51a670bbd58905246f963302c757e4"}
Sep 17 16:42:23.972005 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.971945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db" event={"ID":"c4158398-2987-4869-9185-3d8962e8ee54","Type":"ContainerStarted","Data":"ce4a254ef0a1730f983240909d9a45099a1b2b306d62def4036755a2b29de3a9"}
Sep 17 16:42:23.972005 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.971995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db" event={"ID":"c4158398-2987-4869-9185-3d8962e8ee54","Type":"ContainerDied","Data":"500a1c03f3f58abf89c6649dc1a416a5ad35f1be20b9508bbf27c9c6e10b1366"}
Sep 17 16:42:23.972090 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.972027    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db" event={"ID":"c4158398-2987-4869-9185-3d8962e8ee54","Type":"ContainerStarted","Data":"d4d2e8579763108020297b86f4e6a4829c3c7afee8386d0662569e3fe8c118d8"}
Sep 17 16:42:23.973335 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.973319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rkpps" event={"ID":"d0aa3902-ccf1-49cf-ad48-aafa74d040f0","Type":"ContainerStarted","Data":"d6c55ff1960edc716e1a65e0c70e5d85c12381de5c1b2bda4944a2d8b197af24"}
Sep 17 16:42:23.974706 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.974689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kfplg" event={"ID":"0513c985-c365-41ee-959d-b02794d96aab","Type":"ContainerStarted","Data":"0129b5cebcade911d1b680e766259d2953bd5fdac0f2b4d09ddc6b054d3992b9"}
Sep 17 16:42:23.976170 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.976153    2581 generic.go:356] "Generic (PLEG): container finished" podID="d04dd819-1558-44a9-9fe9-cdb6fd48ec82" containerID="dcfee1b10329e70c68ce45e522667eb3d5466328187bac8393462bf7baf88291" exitCode=0
Sep 17 16:42:23.976239 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.976213    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzjw5" event={"ID":"d04dd819-1558-44a9-9fe9-cdb6fd48ec82","Type":"ContainerDied","Data":"dcfee1b10329e70c68ce45e522667eb3d5466328187bac8393462bf7baf88291"}
Sep 17 16:42:23.977660 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.977636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vscfj" event={"ID":"21fa7670-4703-438a-9c6b-3caa76193e5b","Type":"ContainerStarted","Data":"c95d67c910c6742ed2eef7f784de71ad3e1280a5ddfa9dd29406b357dd55e287"}
Sep 17 16:42:23.988372 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.988337    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2npqv" podStartSLOduration=5.112296416 podStartE2EDuration="21.988325772s" totalImagesPullingTime="16.876029356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:23.987628988 +0000 UTC m=+23.410566862" watchObservedRunningTime="2026-09-17 16:42:23.988325772 +0000 UTC m=+23.411263632"
Sep 17 16:42:23.988460 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:23.988443    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-49.ec2.internal" podStartSLOduration=20.988439494 podStartE2EDuration="20.988439494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:07.954554701 +0000 UTC m=+7.377492566" watchObservedRunningTime="2026-09-17 16:42:23.988439494 +0000 UTC m=+23.411377358"
Sep 17 16:42:24.030412 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:24.030369    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rkpps" podStartSLOduration=5.181535018 podStartE2EDuration="22.030352684s" totalImagesPullingTime="16.848817666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:24.02985811 +0000 UTC m=+23.452795974" watchObservedRunningTime="2026-09-17 16:42:24.030352684 +0000 UTC m=+23.453290548"
Sep 17 16:42:24.047621 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:24.047583    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vscfj" podStartSLOduration=4.614536104 podStartE2EDuration="22.047570773s" totalImagesPullingTime="17.433034669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:24.047029953 +0000 UTC m=+23.469967818" watchObservedRunningTime="2026-09-17 16:42:24.047570773 +0000 UTC m=+23.470508635"
Sep 17 16:42:24.072866 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:24.072825    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mpgnj" podStartSLOduration=5.148088908 podStartE2EDuration="22.072806485s" totalImagesPullingTime="16.924717577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:24.072731569 +0000 UTC m=+23.495669432" watchObservedRunningTime="2026-09-17 16:42:24.072806485 +0000 UTC m=+23.495744348"
Sep 17 16:42:24.550278 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:24.550254    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 16:42:24.773771 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:24.773646    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T16:42:24.550274316Z","UUID":"a182168b-12ae-40fd-b03f-b99a29fc9414","Handler":null,"Name":"","Endpoint":""}
Sep 17 16:42:24.775187 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:24.775167    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 16:42:24.775303 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:24.775198    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 16:42:24.830249 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:24.830220    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:24.830392 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:24.830227    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:24.830392 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:24.830351    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:24.830392 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:24.830346    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:24.830547 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:24.830453    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:24.830605 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:24.830546    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:24.982887 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:24.982855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk" event={"ID":"d5dc52de-ad02-4a17-ab94-629161ff514b","Type":"ContainerStarted","Data":"e4603b4f0650d7cd7839f33d3f17e01e895fe628c1abcce5517a5b2ef4f5e85b"}
Sep 17 16:42:24.984845 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:24.984822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sc2rc" event={"ID":"965e6eb0-6a53-451c-9331-4c48bc36d1cb","Type":"ContainerStarted","Data":"7556763e05bd01632d9da26651837c1667a8701c6006bdb729ff856ad449afc6"}
Sep 17 16:42:25.011187 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:25.011141    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kfplg" podStartSLOduration=6.552164992 podStartE2EDuration="23.011122159s" totalImagesPullingTime="16.458957167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:24.090406758 +0000 UTC m=+23.513344622" watchObservedRunningTime="2026-09-17 16:42:25.011122159 +0000 UTC m=+24.434060026"
Sep 17 16:42:25.011426 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:25.011387    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sc2rc" podStartSLOduration=6.13669829 podStartE2EDuration="23.011375373s" totalImagesPullingTime="16.874677083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:25.010884428 +0000 UTC m=+24.433822292" watchObservedRunningTime="2026-09-17 16:42:25.011375373 +0000 UTC m=+24.434313237"
Sep 17 16:42:25.989857 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:25.989782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk" event={"ID":"d5dc52de-ad02-4a17-ab94-629161ff514b","Type":"ContainerStarted","Data":"ad299990f3c3426e0686bba530f4b0e884f2de322de77e6c54125aed87966c12"}
Sep 17 16:42:25.993666 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:25.993600    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 16:42:25.994067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:25.994042    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db" event={"ID":"c4158398-2987-4869-9185-3d8962e8ee54","Type":"ContainerStarted","Data":"f72fb1f5a43e3cf8f751044ccbbb1f92628db9831d23c4aa28bcab59090f09ff"}
Sep 17 16:42:26.830478 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:26.830414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:26.830478 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:26.830426    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:26.830720 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:26.830451    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:26.830720 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:26.830567    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:26.830720 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:26.830674    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:26.830864 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:26.830767    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:28.755185 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:28.755004    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vscfj"
Sep 17 16:42:28.755937 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:28.755810    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vscfj"
Sep 17 16:42:28.775740 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:28.775698    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ft2gk" podStartSLOduration=7.517433822 podStartE2EDuration="26.77568501s" totalImagesPullingTime="19.258251188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:26.012089366 +0000 UTC m=+25.435027230" watchObservedRunningTime="2026-09-17 16:42:28.77568501 +0000 UTC m=+28.198622873"
Sep 17 16:42:28.830358 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:28.830331    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:28.830456 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:28.830356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:28.830456 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:28.830334    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:28.830547 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:28.830459    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:28.830627 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:28.830542    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:28.830627 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:28.830613    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:29.004504 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:29.004481    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 16:42:29.004929 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:29.004904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db" event={"ID":"c4158398-2987-4869-9185-3d8962e8ee54","Type":"ContainerStarted","Data":"0fbebd21c4006576771b29c8931fb18e2b48a5dfdf3406b2cb8385e459ae9e8f"}
Sep 17 16:42:29.005271 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:29.005182    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:29.005399 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:29.005274    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:29.005399 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:29.005324    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:29.005399 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:29.005353    2581 scope.go:117] "RemoveContainer" containerID="500a1c03f3f58abf89c6649dc1a416a5ad35f1be20b9508bbf27c9c6e10b1366"
Sep 17 16:42:29.008017 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:29.007986    2581 generic.go:356] "Generic (PLEG): container finished" podID="d04dd819-1558-44a9-9fe9-cdb6fd48ec82" containerID="f01e594042ac6900047f37ae35f37c70a79005165ebd3f3bfed8c9be70e6f2a2" exitCode=0
Sep 17 16:42:29.008110 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:29.008022    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzjw5" event={"ID":"d04dd819-1558-44a9-9fe9-cdb6fd48ec82","Type":"ContainerDied","Data":"f01e594042ac6900047f37ae35f37c70a79005165ebd3f3bfed8c9be70e6f2a2"}
Sep 17 16:42:29.008680 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:29.008478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vscfj"
Sep 17 16:42:29.008780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:29.008749    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vscfj"
Sep 17 16:42:29.023601 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:29.023577    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:29.023956 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:29.023937    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:42:30.013442 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:30.013312    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 16:42:30.013810 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:30.013780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db" event={"ID":"c4158398-2987-4869-9185-3d8962e8ee54","Type":"ContainerStarted","Data":"68507834ca888e1b41aeac56b12b715645104032fd0dfbb6f8e51f8a560fd4ea"}
Sep 17 16:42:30.044539 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:30.044494    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db" podStartSLOduration=11.156143343 podStartE2EDuration="28.044479932s" totalImagesPullingTime="16.888336589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:30.043277571 +0000 UTC m=+29.466215434" watchObservedRunningTime="2026-09-17 16:42:30.044479932 +0000 UTC m=+29.467417795"
Sep 17 16:42:30.375276 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:30.375189    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lgk7x"]
Sep 17 16:42:30.375411 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:30.375296    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:30.375411 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:30.375385    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:30.377884 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:30.377855    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-24wdv"]
Sep 17 16:42:30.378011 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:30.377957    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:30.378076 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:30.378058    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:30.393709 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:30.393683    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qz854"]
Sep 17 16:42:30.393788 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:30.393777    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:30.393885 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:30.393854    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:31.018867 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:31.018834    2581 generic.go:356] "Generic (PLEG): container finished" podID="d04dd819-1558-44a9-9fe9-cdb6fd48ec82" containerID="6f700c0ab0db9761848478e054c47970079a6b85c0726c1c8ec6927394b58d1f" exitCode=0
Sep 17 16:42:31.019311 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:31.018924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzjw5" event={"ID":"d04dd819-1558-44a9-9fe9-cdb6fd48ec82","Type":"ContainerDied","Data":"6f700c0ab0db9761848478e054c47970079a6b85c0726c1c8ec6927394b58d1f"}
Sep 17 16:42:31.830055 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:31.830031    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:31.830131 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:31.830059    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:31.830131 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:31.830060    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:31.830197 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:31.830148    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:31.830286 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:31.830258    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:31.830352 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:31.830337    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:32.023797 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:32.023738    2581 generic.go:356] "Generic (PLEG): container finished" podID="d04dd819-1558-44a9-9fe9-cdb6fd48ec82" containerID="df77f0cdd2e83032fdc2ea47e6375a0a01a1ae49c9b8cfbcd440920fcbeabbb6" exitCode=0
Sep 17 16:42:32.024100 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:32.023797    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzjw5" event={"ID":"d04dd819-1558-44a9-9fe9-cdb6fd48ec82","Type":"ContainerDied","Data":"df77f0cdd2e83032fdc2ea47e6375a0a01a1ae49c9b8cfbcd440920fcbeabbb6"}
Sep 17 16:42:33.829733 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:33.829701    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:33.829733 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:33.829736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:33.830249 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:33.829701    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:33.830249 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:33.829813    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:33.830249 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:33.829915    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:33.830249 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:33.830017    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:35.829537 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:35.829345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:35.830015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:35.829396    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:35.830015 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:35.829594    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgk7x" podUID="667acd31-9d7e-4010-bbb3-f18b17232f48"
Sep 17 16:42:35.830015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:35.829396    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:35.830015 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:35.829648    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qz854" podUID="469c498c-7671-4d9b-ae5d-ee75d043b4d4"
Sep 17 16:42:35.830015 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:35.829712    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24wdv" podUID="d306fc98-5114-4b68-a4f7-bfaaf0e419d1"
Sep 17 16:42:35.987189 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:35.987168    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-49.ec2.internal" event="NodeReady"
Sep 17 16:42:35.987318 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:35.987281    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 16:42:36.034499 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.034469    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gqqcv"]
Sep 17 16:42:36.037732 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.037633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:36.038315 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.038288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzjw5" event={"ID":"d04dd819-1558-44a9-9fe9-cdb6fd48ec82","Type":"ContainerStarted","Data":"964d8df5d1e751423cc76229d1110bc3f15830703bf9151b8e836d6ca5dd1c8a"}
Sep 17 16:42:36.040168 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.039885    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 16:42:36.040168 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.040100    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b2jmz\""
Sep 17 16:42:36.040168 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.040109    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 16:42:36.041037 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.041001    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mwc8d"]
Sep 17 16:42:36.044530 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.044511    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.046617 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.046595    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cp4pd\""
Sep 17 16:42:36.046717 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.046630    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 16:42:36.047267 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.047060    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 16:42:36.047267 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.047090    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 16:42:36.047267 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.047115    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 16:42:36.051708 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.051689    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gqqcv"]
Sep 17 16:42:36.058901 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.058879    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mwc8d"]
Sep 17 16:42:36.063832 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.063808    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l6gdl"]
Sep 17 16:42:36.066745 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.066728    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l6gdl"
Sep 17 16:42:36.069015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.068997    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 16:42:36.069167 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.069151    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 16:42:36.069237 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.069173    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h8q8g\""
Sep 17 16:42:36.069382 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.069364    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 16:42:36.079869 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.079817    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l6gdl"]
Sep 17 16:42:36.151765 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.151741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-config-volume\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:36.151849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.151799    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd5p8\" (UniqueName: \"kubernetes.io/projected/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-kube-api-access-hd5p8\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:36.151849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.151816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjk6\" (UniqueName: \"kubernetes.io/projected/a5182664-069c-4e58-acdc-327576520ea1-kube-api-access-jrjk6\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.151960 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.151876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:36.151960 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.151893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-tmp-dir\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:36.151960 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.151906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.152132 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.151955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5182664-069c-4e58-acdc-327576520ea1-data-volume\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.152132 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.152036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a5182664-069c-4e58-acdc-327576520ea1-crio-socket\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.152132 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.152073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.252942 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.252919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:36.253053 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.252946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/328baa78-1822-4052-8b99-72fc548a3ece-cert\") pod \"ingress-canary-l6gdl\" (UID: \"328baa78-1822-4052-8b99-72fc548a3ece\") " pod="openshift-ingress-canary/ingress-canary-l6gdl"
Sep 17 16:42:36.253053 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.252962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-tmp-dir\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:36.253053 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.253003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.253053 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.253035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t42tq\" (UniqueName: \"kubernetes.io/projected/328baa78-1822-4052-8b99-72fc548a3ece-kube-api-access-t42tq\") pod \"ingress-canary-l6gdl\" (UID: \"328baa78-1822-4052-8b99-72fc548a3ece\") " pod="openshift-ingress-canary/ingress-canary-l6gdl"
Sep 17 16:42:36.253268 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.253057    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:36.253268 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.253062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5182664-069c-4e58-acdc-327576520ea1-data-volume\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.253268 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.253108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a5182664-069c-4e58-acdc-327576520ea1-crio-socket\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.253268 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.253116    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:36.253268 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.253134    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls podName:c2fed88c-fffb-4c6f-9532-5af8fccb6b98 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:36.753108171 +0000 UTC m=+36.176046028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls") pod "dns-default-gqqcv" (UID: "c2fed88c-fffb-4c6f-9532-5af8fccb6b98") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:36.253268 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.253182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.253268 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.253235    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:36.253268 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.253243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-tmp-dir\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:36.253268 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.253235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-config-volume\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:36.253766 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.253279    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls podName:a5182664-069c-4e58-acdc-327576520ea1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:36.753263789 +0000 UTC m=+36.176201647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwc8d" (UID: "a5182664-069c-4e58-acdc-327576520ea1") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:36.253766 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.253291    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a5182664-069c-4e58-acdc-327576520ea1-crio-socket\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.253766 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.253309    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm podName:a5182664-069c-4e58-acdc-327576520ea1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:36.753299269 +0000 UTC m=+36.176237117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwc8d" (UID: "a5182664-069c-4e58-acdc-327576520ea1") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:36.253766 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.253340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd5p8\" (UniqueName: \"kubernetes.io/projected/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-kube-api-access-hd5p8\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:36.253766 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.253363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjk6\" (UniqueName: \"kubernetes.io/projected/a5182664-069c-4e58-acdc-327576520ea1-kube-api-access-jrjk6\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.253766 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.253377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5182664-069c-4e58-acdc-327576520ea1-data-volume\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.253766 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.253723    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-config-volume\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:36.271724 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.271696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd5p8\" (UniqueName: \"kubernetes.io/projected/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-kube-api-access-hd5p8\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:36.271804 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.271701    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjk6\" (UniqueName: \"kubernetes.io/projected/a5182664-069c-4e58-acdc-327576520ea1-kube-api-access-jrjk6\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.331266 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.331220    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-zb2fp"]
Sep 17 16:42:36.335084 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.335070    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zb2fp"
Sep 17 16:42:36.337428 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.337413    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 16:42:36.337580 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.337562    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 16:42:36.337646 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.337583    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rmqlx\""
Sep 17 16:42:36.343518 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.343498    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-zb2fp"]
Sep 17 16:42:36.354033 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.354016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/328baa78-1822-4052-8b99-72fc548a3ece-cert\") pod \"ingress-canary-l6gdl\" (UID: \"328baa78-1822-4052-8b99-72fc548a3ece\") " pod="openshift-ingress-canary/ingress-canary-l6gdl"
Sep 17 16:42:36.354129 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.354056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t42tq\" (UniqueName: \"kubernetes.io/projected/328baa78-1822-4052-8b99-72fc548a3ece-kube-api-access-t42tq\") pod \"ingress-canary-l6gdl\" (UID: \"328baa78-1822-4052-8b99-72fc548a3ece\") " pod="openshift-ingress-canary/ingress-canary-l6gdl"
Sep 17 16:42:36.356087 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.356070    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/328baa78-1822-4052-8b99-72fc548a3ece-cert\") pod \"ingress-canary-l6gdl\" (UID: \"328baa78-1822-4052-8b99-72fc548a3ece\") " pod="openshift-ingress-canary/ingress-canary-l6gdl"
Sep 17 16:42:36.366851 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.366818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t42tq\" (UniqueName: \"kubernetes.io/projected/328baa78-1822-4052-8b99-72fc548a3ece-kube-api-access-t42tq\") pod \"ingress-canary-l6gdl\" (UID: \"328baa78-1822-4052-8b99-72fc548a3ece\") " pod="openshift-ingress-canary/ingress-canary-l6gdl"
Sep 17 16:42:36.403814 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.403790    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l6gdl"
Sep 17 16:42:36.454898 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.454874    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs\") pod \"network-metrics-daemon-24wdv\" (UID: \"d306fc98-5114-4b68-a4f7-bfaaf0e419d1\") " pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:36.455019 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.454932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqlzx\" (UniqueName: \"kubernetes.io/projected/b20c7969-d9a4-4323-89de-984436154a25-kube-api-access-vqlzx\") pod \"migrator-76794cbd6c-zb2fp\" (UID: \"b20c7969-d9a4-4323-89de-984436154a25\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zb2fp"
Sep 17 16:42:36.455167 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.455072    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:36.455167 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.455145    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs podName:d306fc98-5114-4b68-a4f7-bfaaf0e419d1 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:08.455124929 +0000 UTC m=+67.878062851 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs") pod "network-metrics-daemon-24wdv" (UID: "d306fc98-5114-4b68-a4f7-bfaaf0e419d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:36.555618 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.555591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqlzx\" (UniqueName: \"kubernetes.io/projected/b20c7969-d9a4-4323-89de-984436154a25-kube-api-access-vqlzx\") pod \"migrator-76794cbd6c-zb2fp\" (UID: \"b20c7969-d9a4-4323-89de-984436154a25\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zb2fp"
Sep 17 16:42:36.555736 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.555649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:36.555776 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.555752    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:36.555831 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.555820    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret podName:667acd31-9d7e-4010-bbb3-f18b17232f48 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:08.555802265 +0000 UTC m=+67.978740129 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret") pod "global-pull-secret-syncer-lgk7x" (UID: "667acd31-9d7e-4010-bbb3-f18b17232f48") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:36.569061 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.569039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqlzx\" (UniqueName: \"kubernetes.io/projected/b20c7969-d9a4-4323-89de-984436154a25-kube-api-access-vqlzx\") pod \"migrator-76794cbd6c-zb2fp\" (UID: \"b20c7969-d9a4-4323-89de-984436154a25\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zb2fp"
Sep 17 16:42:36.643628 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.643576    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zb2fp"
Sep 17 16:42:36.656993 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.656957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmc9z\" (UniqueName: \"kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z\") pod \"network-check-target-qz854\" (UID: \"469c498c-7671-4d9b-ae5d-ee75d043b4d4\") " pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:36.657126 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.657106    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:36.657168 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.657133    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:36.657168 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.657147    2581 projected.go:196] Error preparing data for projected volume kube-api-access-kmc9z for pod openshift-network-diagnostics/network-check-target-qz854: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:36.657232 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.657208    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z podName:469c498c-7671-4d9b-ae5d-ee75d043b4d4 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:08.657190455 +0000 UTC m=+68.080128315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kmc9z" (UniqueName: "kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z") pod "network-check-target-qz854" (UID: "469c498c-7671-4d9b-ae5d-ee75d043b4d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:36.684949 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.684920    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l6gdl"]
Sep 17 16:42:36.688211 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:36.688185    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328baa78_1822_4052_8b99_72fc548a3ece.slice/crio-4f8984ecaea5f11503725f9ad580877ccafaf169758a63b59127230abee0c5ef WatchSource:0}: Error finding container 4f8984ecaea5f11503725f9ad580877ccafaf169758a63b59127230abee0c5ef: Status 404 returned error can't find the container with id 4f8984ecaea5f11503725f9ad580877ccafaf169758a63b59127230abee0c5ef
Sep 17 16:42:36.757397 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.757369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.757497 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.757444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:36.757497 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.757470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:36.757580 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.757515    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:36.757580 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.757562    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:36.757580 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.757579    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls podName:a5182664-069c-4e58-acdc-327576520ea1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:37.757564428 +0000 UTC m=+37.180502283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwc8d" (UID: "a5182664-069c-4e58-acdc-327576520ea1") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:36.757689 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.757608    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls podName:c2fed88c-fffb-4c6f-9532-5af8fccb6b98 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:37.757595726 +0000 UTC m=+37.180533586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls") pod "dns-default-gqqcv" (UID: "c2fed88c-fffb-4c6f-9532-5af8fccb6b98") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:36.757689 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.757616    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:36.757689 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:36.757658    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm podName:a5182664-069c-4e58-acdc-327576520ea1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:37.757647293 +0000 UTC m=+37.180585135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwc8d" (UID: "a5182664-069c-4e58-acdc-327576520ea1") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:36.916379 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:36.916329    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-zb2fp"]
Sep 17 16:42:36.918995 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:36.918956    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20c7969_d9a4_4323_89de_984436154a25.slice/crio-32706c4a467aea3f5f9d86fd8c9acc420727441446b62cd38618d27746b436d5 WatchSource:0}: Error finding container 32706c4a467aea3f5f9d86fd8c9acc420727441446b62cd38618d27746b436d5: Status 404 returned error can't find the container with id 32706c4a467aea3f5f9d86fd8c9acc420727441446b62cd38618d27746b436d5
Sep 17 16:42:37.041324 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.041302    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zb2fp" event={"ID":"b20c7969-d9a4-4323-89de-984436154a25","Type":"ContainerStarted","Data":"32706c4a467aea3f5f9d86fd8c9acc420727441446b62cd38618d27746b436d5"}
Sep 17 16:42:37.043997 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.043959    2581 generic.go:356] "Generic (PLEG): container finished" podID="d04dd819-1558-44a9-9fe9-cdb6fd48ec82" containerID="964d8df5d1e751423cc76229d1110bc3f15830703bf9151b8e836d6ca5dd1c8a" exitCode=0
Sep 17 16:42:37.044090 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.044055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzjw5" event={"ID":"d04dd819-1558-44a9-9fe9-cdb6fd48ec82","Type":"ContainerDied","Data":"964d8df5d1e751423cc76229d1110bc3f15830703bf9151b8e836d6ca5dd1c8a"}
Sep 17 16:42:37.045331 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.045299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l6gdl" event={"ID":"328baa78-1822-4052-8b99-72fc548a3ece","Type":"ContainerStarted","Data":"4f8984ecaea5f11503725f9ad580877ccafaf169758a63b59127230abee0c5ef"}
Sep 17 16:42:37.464772 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.464738    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rkpps_d0aa3902-ccf1-49cf-ad48-aafa74d040f0/dns-node-resolver/0.log"
Sep 17 16:42:37.766072 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.765994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:37.766072 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.766044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:37.766262 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.766114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:37.766262 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:37.766132    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:37.766262 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:37.766172    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:37.766262 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:37.766212    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls podName:c2fed88c-fffb-4c6f-9532-5af8fccb6b98 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:39.766191265 +0000 UTC m=+39.189129128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls") pod "dns-default-gqqcv" (UID: "c2fed88c-fffb-4c6f-9532-5af8fccb6b98") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:37.766262 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:37.766216    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:37.766262 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:37.766232    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm podName:a5182664-069c-4e58-acdc-327576520ea1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:39.766221117 +0000 UTC m=+39.189158958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwc8d" (UID: "a5182664-069c-4e58-acdc-327576520ea1") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:37.766498 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:37.766277    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls podName:a5182664-069c-4e58-acdc-327576520ea1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:39.76626086 +0000 UTC m=+39.189198715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwc8d" (UID: "a5182664-069c-4e58-acdc-327576520ea1") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:37.830378 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.829829    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:42:37.830378 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.829836    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:42:37.830378 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.829839    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:42:37.833588 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.833563    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 16:42:37.833731 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.833593    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k782z\""
Sep 17 16:42:37.833731 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.833632    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 16:42:37.833931 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.833763    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hqd8r\""
Sep 17 16:42:37.833931 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.833846    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 16:42:37.834164 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:37.833985    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 16:42:38.052019 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.051990    2581 generic.go:356] "Generic (PLEG): container finished" podID="d04dd819-1558-44a9-9fe9-cdb6fd48ec82" containerID="2ab15d168188f78f8201914ae6b1580a5e57bf1e49a6803d06b9c94bc75bd502" exitCode=0
Sep 17 16:42:38.052375 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.052068    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzjw5" event={"ID":"d04dd819-1558-44a9-9fe9-cdb6fd48ec82","Type":"ContainerDied","Data":"2ab15d168188f78f8201914ae6b1580a5e57bf1e49a6803d06b9c94bc75bd502"}
Sep 17 16:42:38.226584 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.226555    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-9zpjh"]
Sep 17 16:42:38.229468 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.229453    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-9zpjh"
Sep 17 16:42:38.231871 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.231844    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 16:42:38.231958 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.231846    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dwld6\""
Sep 17 16:42:38.232367 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.232343    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 16:42:38.232486 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.232351    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 16:42:38.232858 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.232822    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 16:42:38.243042 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.243022    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-9zpjh"]
Sep 17 16:42:38.373891 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.373821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t625g\" (UniqueName: \"kubernetes.io/projected/59ac9b0c-8840-496f-825e-b52bbb338f13-kube-api-access-t625g\") pod \"service-ca-55f6fc7b86-9zpjh\" (UID: \"59ac9b0c-8840-496f-825e-b52bbb338f13\") " pod="openshift-service-ca/service-ca-55f6fc7b86-9zpjh"
Sep 17 16:42:38.374016 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.373890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/59ac9b0c-8840-496f-825e-b52bbb338f13-signing-cabundle\") pod \"service-ca-55f6fc7b86-9zpjh\" (UID: \"59ac9b0c-8840-496f-825e-b52bbb338f13\") " pod="openshift-service-ca/service-ca-55f6fc7b86-9zpjh"
Sep 17 16:42:38.374016 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.373920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/59ac9b0c-8840-496f-825e-b52bbb338f13-signing-key\") pod \"service-ca-55f6fc7b86-9zpjh\" (UID: \"59ac9b0c-8840-496f-825e-b52bbb338f13\") " pod="openshift-service-ca/service-ca-55f6fc7b86-9zpjh"
Sep 17 16:42:38.447262 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.447238    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kfplg_0513c985-c365-41ee-959d-b02794d96aab/node-ca/0.log"
Sep 17 16:42:38.474774 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.474753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/59ac9b0c-8840-496f-825e-b52bbb338f13-signing-cabundle\") pod \"service-ca-55f6fc7b86-9zpjh\" (UID: \"59ac9b0c-8840-496f-825e-b52bbb338f13\") " pod="openshift-service-ca/service-ca-55f6fc7b86-9zpjh"
Sep 17 16:42:38.474885 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.474794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/59ac9b0c-8840-496f-825e-b52bbb338f13-signing-key\") pod \"service-ca-55f6fc7b86-9zpjh\" (UID: \"59ac9b0c-8840-496f-825e-b52bbb338f13\") " pod="openshift-service-ca/service-ca-55f6fc7b86-9zpjh"
Sep 17 16:42:38.474930 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.474894    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t625g\" (UniqueName: \"kubernetes.io/projected/59ac9b0c-8840-496f-825e-b52bbb338f13-kube-api-access-t625g\") pod \"service-ca-55f6fc7b86-9zpjh\" (UID: \"59ac9b0c-8840-496f-825e-b52bbb338f13\") " pod="openshift-service-ca/service-ca-55f6fc7b86-9zpjh"
Sep 17 16:42:38.475444 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.475426    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/59ac9b0c-8840-496f-825e-b52bbb338f13-signing-cabundle\") pod \"service-ca-55f6fc7b86-9zpjh\" (UID: \"59ac9b0c-8840-496f-825e-b52bbb338f13\") " pod="openshift-service-ca/service-ca-55f6fc7b86-9zpjh"
Sep 17 16:42:38.478409 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.478391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/59ac9b0c-8840-496f-825e-b52bbb338f13-signing-key\") pod \"service-ca-55f6fc7b86-9zpjh\" (UID: \"59ac9b0c-8840-496f-825e-b52bbb338f13\") " pod="openshift-service-ca/service-ca-55f6fc7b86-9zpjh"
Sep 17 16:42:38.485305 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.485284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t625g\" (UniqueName: \"kubernetes.io/projected/59ac9b0c-8840-496f-825e-b52bbb338f13-kube-api-access-t625g\") pod \"service-ca-55f6fc7b86-9zpjh\" (UID: \"59ac9b0c-8840-496f-825e-b52bbb338f13\") " pod="openshift-service-ca/service-ca-55f6fc7b86-9zpjh"
Sep 17 16:42:38.540209 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.540183    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-9zpjh"
Sep 17 16:42:38.825604 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:38.825412    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-9zpjh"]
Sep 17 16:42:38.829126 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:38.829094    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ac9b0c_8840_496f_825e_b52bbb338f13.slice/crio-34fde7d12cde946718012dab1e864c71716440b75eea3ffbbce6e36a008d39b1 WatchSource:0}: Error finding container 34fde7d12cde946718012dab1e864c71716440b75eea3ffbbce6e36a008d39b1: Status 404 returned error can't find the container with id 34fde7d12cde946718012dab1e864c71716440b75eea3ffbbce6e36a008d39b1
Sep 17 16:42:39.059260 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:39.056802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zb2fp" event={"ID":"b20c7969-d9a4-4323-89de-984436154a25","Type":"ContainerStarted","Data":"5a0ca8690893b39748d6dc9137bb2fb73f73219a5d036b54c8a807dc3433e3c8"}
Sep 17 16:42:39.059260 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:39.056935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zb2fp" event={"ID":"b20c7969-d9a4-4323-89de-984436154a25","Type":"ContainerStarted","Data":"6b798971c87d73bd6ac665f8eb1bf557abdd951ce39e0caa584451c2c907f776"}
Sep 17 16:42:39.061562 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:39.061534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-9zpjh" event={"ID":"59ac9b0c-8840-496f-825e-b52bbb338f13","Type":"ContainerStarted","Data":"34fde7d12cde946718012dab1e864c71716440b75eea3ffbbce6e36a008d39b1"}
Sep 17 16:42:39.065589 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:39.065566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dzjw5" event={"ID":"d04dd819-1558-44a9-9fe9-cdb6fd48ec82","Type":"ContainerStarted","Data":"939f6152713f94492dbe4e87eebeb4b9b41d7c4cbcd2f3e57d74c540a18d4418"}
Sep 17 16:42:39.079630 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:39.079330    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zb2fp" podStartSLOduration=2.012415541 podStartE2EDuration="3.079315113s" totalImagesPullingTime="1.066899572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:39.079062572 +0000 UTC m=+38.502000447" watchObservedRunningTime="2026-09-17 16:42:39.079315113 +0000 UTC m=+38.502252987"
Sep 17 16:42:39.107884 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:39.107836    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dzjw5" podStartSLOduration=9.500377392 podStartE2EDuration="37.107804174s" totalImagesPullingTime="27.607426782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:39.106665211 +0000 UTC m=+38.529603078" watchObservedRunningTime="2026-09-17 16:42:39.107804174 +0000 UTC m=+38.530742037"
Sep 17 16:42:39.786045 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:39.786008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:39.786045 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:39.786048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:39.786237 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:39.786106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:39.786237 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:39.786162    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:39.786237 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:39.786199    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 16:42:39.786237 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:39.786210    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:39.786358 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:39.786244    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls podName:c2fed88c-fffb-4c6f-9532-5af8fccb6b98 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:43.786226902 +0000 UTC m=+43.209164748 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls") pod "dns-default-gqqcv" (UID: "c2fed88c-fffb-4c6f-9532-5af8fccb6b98") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:39.786358 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:39.786261    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls podName:a5182664-069c-4e58-acdc-327576520ea1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:43.786249901 +0000 UTC m=+43.209187779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwc8d" (UID: "a5182664-069c-4e58-acdc-327576520ea1") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:39.786358 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:39.786275    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm podName:a5182664-069c-4e58-acdc-327576520ea1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:43.786268681 +0000 UTC m=+43.209206521 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mwc8d" (UID: "a5182664-069c-4e58-acdc-327576520ea1") : configmap "kube-rbac-proxy" not found
Sep 17 16:42:41.073619 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:41.073587    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l6gdl" event={"ID":"328baa78-1822-4052-8b99-72fc548a3ece","Type":"ContainerStarted","Data":"8b2a77d149db537e9e2531d9bf5eb03569b08682e4d2c11028ef0a9051fb4132"}
Sep 17 16:42:42.078175 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:42.078105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-9zpjh" event={"ID":"59ac9b0c-8840-496f-825e-b52bbb338f13","Type":"ContainerStarted","Data":"5f9963013638dc7899f6e001f86b25a01de43b019db0123ac06019a6dc588fae"}
Sep 17 16:42:42.096623 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:42.096576    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l6gdl" podStartSLOduration=2.7515427199999998 podStartE2EDuration="6.09655695s" totalImagesPullingTime="3.34501423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:41.097684297 +0000 UTC m=+40.520622161" watchObservedRunningTime="2026-09-17 16:42:42.09655695 +0000 UTC m=+41.519494813"
Sep 17 16:42:42.096824 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:42.096672    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-9zpjh" podStartSLOduration=1.118645275 podStartE2EDuration="4.096664186s" totalImagesPullingTime="2.978018911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:42.096017719 +0000 UTC m=+41.518955581" watchObservedRunningTime="2026-09-17 16:42:42.096664186 +0000 UTC m=+41.519602051"
Sep 17 16:42:43.816880 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:43.816845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:43.817342 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:43.816890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:43.817342 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:43.816952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:43.817342 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:43.817028    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 16:42:43.817342 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:43.817092    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:43.817342 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:43.817102    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls podName:c2fed88c-fffb-4c6f-9532-5af8fccb6b98 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:51.81707859 +0000 UTC m=+51.240016449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls") pod "dns-default-gqqcv" (UID: "c2fed88c-fffb-4c6f-9532-5af8fccb6b98") : secret "dns-default-metrics-tls" not found
Sep 17 16:42:43.817342 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:42:43.817139    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls podName:a5182664-069c-4e58-acdc-327576520ea1 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:51.817123829 +0000 UTC m=+51.240061685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mwc8d" (UID: "a5182664-069c-4e58-acdc-327576520ea1") : secret "insights-runtime-extractor-tls" not found
Sep 17 16:42:43.830564 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:43.830539    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a5182664-069c-4e58-acdc-327576520ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:51.870502 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:51.870467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:51.870932 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:51.870523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:51.873413 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:51.873386    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a5182664-069c-4e58-acdc-327576520ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwc8d\" (UID: \"a5182664-069c-4e58-acdc-327576520ea1\") " pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:51.873527 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:51.873438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2fed88c-fffb-4c6f-9532-5af8fccb6b98-metrics-tls\") pod \"dns-default-gqqcv\" (UID: \"c2fed88c-fffb-4c6f-9532-5af8fccb6b98\") " pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:51.990906 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:51.990882    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:51.999042 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:51.999022    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mwc8d"
Sep 17 16:42:52.291867 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:52.291726    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gqqcv"]
Sep 17 16:42:52.294213 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:52.294187    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2fed88c_fffb_4c6f_9532_5af8fccb6b98.slice/crio-a371a8901235be48f50a831c596eff810b5d05cb8d9ea514fc18d9e13ff9f41d WatchSource:0}: Error finding container a371a8901235be48f50a831c596eff810b5d05cb8d9ea514fc18d9e13ff9f41d: Status 404 returned error can't find the container with id a371a8901235be48f50a831c596eff810b5d05cb8d9ea514fc18d9e13ff9f41d
Sep 17 16:42:52.322096 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:52.322073    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mwc8d"]
Sep 17 16:42:52.325391 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:42:52.325366    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5182664_069c_4e58_acdc_327576520ea1.slice/crio-9240acb7989b6e1a6c9a6fea386d4bd1d38b0d629dd511a43df0210d10fb2b3a WatchSource:0}: Error finding container 9240acb7989b6e1a6c9a6fea386d4bd1d38b0d629dd511a43df0210d10fb2b3a: Status 404 returned error can't find the container with id 9240acb7989b6e1a6c9a6fea386d4bd1d38b0d629dd511a43df0210d10fb2b3a
Sep 17 16:42:53.107314 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:53.107268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwc8d" event={"ID":"a5182664-069c-4e58-acdc-327576520ea1","Type":"ContainerStarted","Data":"12b6878086e3dabe3f64a7389cbf4d36d374e5a3f075e086098dd008e127bafd"}
Sep 17 16:42:53.107734 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:53.107378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwc8d" event={"ID":"a5182664-069c-4e58-acdc-327576520ea1","Type":"ContainerStarted","Data":"9240acb7989b6e1a6c9a6fea386d4bd1d38b0d629dd511a43df0210d10fb2b3a"}
Sep 17 16:42:53.108503 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:53.108465    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gqqcv" event={"ID":"c2fed88c-fffb-4c6f-9532-5af8fccb6b98","Type":"ContainerStarted","Data":"a371a8901235be48f50a831c596eff810b5d05cb8d9ea514fc18d9e13ff9f41d"}
Sep 17 16:42:55.118025 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:55.117989    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwc8d" event={"ID":"a5182664-069c-4e58-acdc-327576520ea1","Type":"ContainerStarted","Data":"ff0e3e99d3a9a1a3fba267dfe95922be7b443529a9c708c210c93fa861ce78b2"}
Sep 17 16:42:55.120380 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:55.120300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gqqcv" event={"ID":"c2fed88c-fffb-4c6f-9532-5af8fccb6b98","Type":"ContainerStarted","Data":"3b08ca6607de37e062a51dc30b355ccd14e85a30cd44ea9406b69b55a3917c39"}
Sep 17 16:42:55.120380 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:55.120348    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gqqcv" event={"ID":"c2fed88c-fffb-4c6f-9532-5af8fccb6b98","Type":"ContainerStarted","Data":"811d420b8963ec8118261e74801e591fa6a9defeb8ef1f198b25a2604b6d8ad4"}
Sep 17 16:42:55.120574 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:55.120442    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:42:55.138779 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:55.138734    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gqqcv" podStartSLOduration=17.056210023 podStartE2EDuration="19.138719021s" totalImagesPullingTime="2.082508998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:55.137305937 +0000 UTC m=+54.560243799" watchObservedRunningTime="2026-09-17 16:42:55.138719021 +0000 UTC m=+54.561656884"
Sep 17 16:42:57.130033 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:57.129993    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwc8d" event={"ID":"a5182664-069c-4e58-acdc-327576520ea1","Type":"ContainerStarted","Data":"167c0ca3c6db71c6a91f6b79be2f69ddac0702c51360f9360dd97094f0b11e82"}
Sep 17 16:42:57.150790 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:42:57.150742    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mwc8d" podStartSLOduration=17.42652619 podStartE2EDuration="21.150726435s" totalImagesPullingTime="3.724200245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:57.14931529 +0000 UTC m=+56.572253152" watchObservedRunningTime="2026-09-17 16:42:57.150726435 +0000 UTC m=+56.573664298"
Sep 17 16:43:01.030987 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:01.030941    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tr5db"
Sep 17 16:43:02.459677 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.459643    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"]
Sep 17 16:43:02.462241 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.462222    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"
Sep 17 16:43:02.465556 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.465530    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 16:43:02.466663 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.466642    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 16:43:02.466792 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.466671    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 16:43:02.467552 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.467533    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 16:43:02.481772 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.481738    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"]
Sep 17 16:43:02.540079 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.540058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8sqt\" (UniqueName: \"kubernetes.io/projected/582fad66-8507-413f-bfcc-5b6917d21e64-kube-api-access-r8sqt\") pod \"klusterlet-addon-workmgr-65544fff6b-gs55z\" (UID: \"582fad66-8507-413f-bfcc-5b6917d21e64\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"
Sep 17 16:43:02.540182 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.540094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/582fad66-8507-413f-bfcc-5b6917d21e64-klusterlet-config\") pod \"klusterlet-addon-workmgr-65544fff6b-gs55z\" (UID: \"582fad66-8507-413f-bfcc-5b6917d21e64\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"
Sep 17 16:43:02.540182 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.540125    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/582fad66-8507-413f-bfcc-5b6917d21e64-tmp\") pod \"klusterlet-addon-workmgr-65544fff6b-gs55z\" (UID: \"582fad66-8507-413f-bfcc-5b6917d21e64\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"
Sep 17 16:43:02.587816 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.587797    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-6h4np"]
Sep 17 16:43:02.589749 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.589734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-6h4np"
Sep 17 16:43:02.592443 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.592424    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 16:43:02.593031 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.593012    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 16:43:02.593468 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.593454    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6p9f5\""
Sep 17 16:43:02.602695 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.602675    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-695b7bc4c4-r96j8"]
Sep 17 16:43:02.604502 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.604484    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.606317 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.606293    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-6h4np"]
Sep 17 16:43:02.607069 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.607046    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 16:43:02.607356 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.607336    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7s58x\""
Sep 17 16:43:02.607434 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.607362    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 16:43:02.609189 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.609169    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 16:43:02.612987 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.612949    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 16:43:02.614431 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.614417    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 16:43:02.635776 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.635755    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-695b7bc4c4-r96j8"]
Sep 17 16:43:02.641088 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.641068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/582fad66-8507-413f-bfcc-5b6917d21e64-klusterlet-config\") pod \"klusterlet-addon-workmgr-65544fff6b-gs55z\" (UID: \"582fad66-8507-413f-bfcc-5b6917d21e64\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"
Sep 17 16:43:02.641173 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.641095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-serving-cert\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.641173 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.641113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-oauth-config\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.641173 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.641132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/582fad66-8507-413f-bfcc-5b6917d21e64-tmp\") pod \"klusterlet-addon-workmgr-65544fff6b-gs55z\" (UID: \"582fad66-8507-413f-bfcc-5b6917d21e64\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"
Sep 17 16:43:02.641173 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.641148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg5fg\" (UniqueName: \"kubernetes.io/projected/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-kube-api-access-tg5fg\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.641358 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.641188    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-service-ca\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.641358 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.641272    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7cjw\" (UniqueName: \"kubernetes.io/projected/4eb30ced-c505-47e5-aca7-abbe51e37473-kube-api-access-b7cjw\") pod \"downloads-6594f4c75-6h4np\" (UID: \"4eb30ced-c505-47e5-aca7-abbe51e37473\") " pod="openshift-console/downloads-6594f4c75-6h4np"
Sep 17 16:43:02.641358 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.641311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-config\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.641358 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.641338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-oauth-serving-cert\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.641550 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.641472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8sqt\" (UniqueName: \"kubernetes.io/projected/582fad66-8507-413f-bfcc-5b6917d21e64-kube-api-access-r8sqt\") pod \"klusterlet-addon-workmgr-65544fff6b-gs55z\" (UID: \"582fad66-8507-413f-bfcc-5b6917d21e64\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"
Sep 17 16:43:02.641550 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.641480    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/582fad66-8507-413f-bfcc-5b6917d21e64-tmp\") pod \"klusterlet-addon-workmgr-65544fff6b-gs55z\" (UID: \"582fad66-8507-413f-bfcc-5b6917d21e64\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"
Sep 17 16:43:02.643338 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.643317    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/582fad66-8507-413f-bfcc-5b6917d21e64-klusterlet-config\") pod \"klusterlet-addon-workmgr-65544fff6b-gs55z\" (UID: \"582fad66-8507-413f-bfcc-5b6917d21e64\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"
Sep 17 16:43:02.667219 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.667195    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 16:43:02.678063 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.678045    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 16:43:02.688041 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.688020    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-555b8df8cf-7hc5c"]
Sep 17 16:43:02.688041 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.688030    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8sqt\" (UniqueName: \"kubernetes.io/projected/582fad66-8507-413f-bfcc-5b6917d21e64-kube-api-access-r8sqt\") pod \"klusterlet-addon-workmgr-65544fff6b-gs55z\" (UID: \"582fad66-8507-413f-bfcc-5b6917d21e64\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"
Sep 17 16:43:02.690675 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.690659    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.693204 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.693186    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 16:43:02.693299 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.693225    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 16:43:02.693299 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.693287    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 16:43:02.695570 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.695552    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qdwd2\""
Sep 17 16:43:02.699703 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.699685    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 16:43:02.719394 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.719336    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-555b8df8cf-7hc5c"]
Sep 17 16:43:02.741927 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.741890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-oauth-config\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.742039 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.741935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-service-ca\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.742039 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.741959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7cjw\" (UniqueName: \"kubernetes.io/projected/4eb30ced-c505-47e5-aca7-abbe51e37473-kube-api-access-b7cjw\") pod \"downloads-6594f4c75-6h4np\" (UID: \"4eb30ced-c505-47e5-aca7-abbe51e37473\") " pod="openshift-console/downloads-6594f4c75-6h4np"
Sep 17 16:43:02.742039 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.741997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg5fg\" (UniqueName: \"kubernetes.io/projected/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-kube-api-access-tg5fg\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.742039 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.742020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-registry-tls\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.742039 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.742039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-oauth-serving-cert\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.742268 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.742075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-trusted-ca\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.742268 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.742103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-installation-pull-secrets\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.742268 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.742144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-bound-sa-token\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.742268 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.742203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-serving-cert\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.742268 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.742240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-registry-certificates\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.742453 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.742277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-ca-trust-extracted\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.742453 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.742301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-config\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.742453 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.742349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzvjq\" (UniqueName: \"kubernetes.io/projected/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-kube-api-access-nzvjq\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.742453 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.742374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-image-registry-private-configuration\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.746382 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.746364    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 16:43:02.746537 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.746522    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 16:43:02.746656 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.746637    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 16:43:02.746773 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.746756    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 16:43:02.746849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.746833    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 16:43:02.753343 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.753322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-config\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.753432 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.753416    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-oauth-serving-cert\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.753472 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.753415    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-service-ca\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.755159 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.755137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-oauth-config\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.755236 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.755143    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-serving-cert\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.756039 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.756025    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 16:43:02.766057 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.766041    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 16:43:02.775042 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.775014    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 16:43:02.776327 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.776306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7cjw\" (UniqueName: \"kubernetes.io/projected/4eb30ced-c505-47e5-aca7-abbe51e37473-kube-api-access-b7cjw\") pod \"downloads-6594f4c75-6h4np\" (UID: \"4eb30ced-c505-47e5-aca7-abbe51e37473\") " pod="openshift-console/downloads-6594f4c75-6h4np"
Sep 17 16:43:02.777538 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.777511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg5fg\" (UniqueName: \"kubernetes.io/projected/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-kube-api-access-tg5fg\") pod \"console-695b7bc4c4-r96j8\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") " pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:02.783774 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.783756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"
Sep 17 16:43:02.843063 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.843035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-ca-trust-extracted\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.843190 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.843085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzvjq\" (UniqueName: \"kubernetes.io/projected/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-kube-api-access-nzvjq\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.843190 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.843109    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-image-registry-private-configuration\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.843190 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.843158    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-registry-tls\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.843355 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.843206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-trusted-ca\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.843355 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.843229    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-installation-pull-secrets\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.843355 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.843271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-bound-sa-token\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.843355 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.843302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-registry-certificates\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.844325 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.844299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-ca-trust-extracted\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.844698 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.844670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-registry-certificates\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.844941 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.844896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-trusted-ca\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.846259 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.846239    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-registry-tls\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.846980 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.846925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-installation-pull-secrets\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.847078 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.847048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-image-registry-private-configuration\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.860538 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.860511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-bound-sa-token\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.860691 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.860672    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzvjq\" (UniqueName: \"kubernetes.io/projected/4fe9c5b4-7dd2-4c5e-acac-531a4aecf301-kube-api-access-nzvjq\") pod \"image-registry-555b8df8cf-7hc5c\" (UID: \"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301\") " pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:02.899795 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.899775    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6p9f5\""
Sep 17 16:43:02.908785 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.908770    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-6h4np"
Sep 17 16:43:02.916148 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.916125    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7s58x\""
Sep 17 16:43:02.924405 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:02.924384    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:03.000709 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:03.000683    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:03.064108 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:03.063785    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"]
Sep 17 16:43:03.154084 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:03.154048    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z" event={"ID":"582fad66-8507-413f-bfcc-5b6917d21e64","Type":"ContainerStarted","Data":"92d305261ebd91c10c2ebd2c6dfd295433d8e7fed8f95ab79713c76ff7cd3ff4"}
Sep 17 16:43:03.194935 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:03.194911    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-6h4np"]
Sep 17 16:43:03.198778 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:43:03.198756    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb30ced_c505_47e5_aca7_abbe51e37473.slice/crio-751af55f0a49d6a74196b19c1e5b448e10e227c4fa3ee85ef4c0edd0e3c7ed66 WatchSource:0}: Error finding container 751af55f0a49d6a74196b19c1e5b448e10e227c4fa3ee85ef4c0edd0e3c7ed66: Status 404 returned error can't find the container with id 751af55f0a49d6a74196b19c1e5b448e10e227c4fa3ee85ef4c0edd0e3c7ed66
Sep 17 16:43:03.209584 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:03.209563    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-695b7bc4c4-r96j8"]
Sep 17 16:43:03.212046 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:43:03.212026    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6aa0fa_a25e_4a5c_a200_cb2e1b146b0b.slice/crio-eb9f9c69cd80d6372dea02e6f8ad85e5adc42e899bc39fc7ad01a86968610a06 WatchSource:0}: Error finding container eb9f9c69cd80d6372dea02e6f8ad85e5adc42e899bc39fc7ad01a86968610a06: Status 404 returned error can't find the container with id eb9f9c69cd80d6372dea02e6f8ad85e5adc42e899bc39fc7ad01a86968610a06
Sep 17 16:43:03.275701 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:03.275657    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-555b8df8cf-7hc5c"]
Sep 17 16:43:03.278312 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:43:03.278292    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe9c5b4_7dd2_4c5e_acac_531a4aecf301.slice/crio-866c8d25f0ffa5dec23d040b1f5418df519ad89d512764b66787fd048170b2dd WatchSource:0}: Error finding container 866c8d25f0ffa5dec23d040b1f5418df519ad89d512764b66787fd048170b2dd: Status 404 returned error can't find the container with id 866c8d25f0ffa5dec23d040b1f5418df519ad89d512764b66787fd048170b2dd
Sep 17 16:43:04.166917 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:04.166879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-6h4np" event={"ID":"4eb30ced-c505-47e5-aca7-abbe51e37473","Type":"ContainerStarted","Data":"751af55f0a49d6a74196b19c1e5b448e10e227c4fa3ee85ef4c0edd0e3c7ed66"}
Sep 17 16:43:04.170024 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:04.169994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c" event={"ID":"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301","Type":"ContainerStarted","Data":"16468349e6b367e999fe23fc2fa674fa358c2fc9614859bbe329d70bf19347c5"}
Sep 17 16:43:04.170158 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:04.170045    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c" event={"ID":"4fe9c5b4-7dd2-4c5e-acac-531a4aecf301","Type":"ContainerStarted","Data":"866c8d25f0ffa5dec23d040b1f5418df519ad89d512764b66787fd048170b2dd"}
Sep 17 16:43:04.170771 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:04.170751    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:04.176243 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:04.176191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695b7bc4c4-r96j8" event={"ID":"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b","Type":"ContainerStarted","Data":"eb9f9c69cd80d6372dea02e6f8ad85e5adc42e899bc39fc7ad01a86968610a06"}
Sep 17 16:43:04.193874 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:04.193210    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c" podStartSLOduration=2.193192404 podStartE2EDuration="2.193192404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:04.191897467 +0000 UTC m=+63.614835331" watchObservedRunningTime="2026-09-17 16:43:04.193192404 +0000 UTC m=+63.616130264"
Sep 17 16:43:05.128703 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:05.127394    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gqqcv"
Sep 17 16:43:08.020381 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.020099    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"]
Sep 17 16:43:08.043543 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.043516    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"]
Sep 17 16:43:08.043714 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.043616    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"
Sep 17 16:43:08.046440 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.046241    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 16:43:08.046959 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.046937    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 16:43:08.047079 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.046952    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 16:43:08.047568 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.047207    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zbpsj\""
Sep 17 16:43:08.047568 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.047294    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 16:43:08.047568 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.047416    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 16:43:08.195859 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.195830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/70256eb4-eec2-4d39-8420-5403098c7cbf-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-5lwqk\" (UID: \"70256eb4-eec2-4d39-8420-5403098c7cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"
Sep 17 16:43:08.196038 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.195863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqdx5\" (UniqueName: \"kubernetes.io/projected/70256eb4-eec2-4d39-8420-5403098c7cbf-kube-api-access-kqdx5\") pod \"prometheus-operator-7bd5d56d6-5lwqk\" (UID: \"70256eb4-eec2-4d39-8420-5403098c7cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"
Sep 17 16:43:08.196038 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.195897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70256eb4-eec2-4d39-8420-5403098c7cbf-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-5lwqk\" (UID: \"70256eb4-eec2-4d39-8420-5403098c7cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"
Sep 17 16:43:08.196186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.196049    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70256eb4-eec2-4d39-8420-5403098c7cbf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-5lwqk\" (UID: \"70256eb4-eec2-4d39-8420-5403098c7cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"
Sep 17 16:43:08.297337 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.297269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70256eb4-eec2-4d39-8420-5403098c7cbf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-5lwqk\" (UID: \"70256eb4-eec2-4d39-8420-5403098c7cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"
Sep 17 16:43:08.297337 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.297325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/70256eb4-eec2-4d39-8420-5403098c7cbf-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-5lwqk\" (UID: \"70256eb4-eec2-4d39-8420-5403098c7cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"
Sep 17 16:43:08.297548 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.297351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqdx5\" (UniqueName: \"kubernetes.io/projected/70256eb4-eec2-4d39-8420-5403098c7cbf-kube-api-access-kqdx5\") pod \"prometheus-operator-7bd5d56d6-5lwqk\" (UID: \"70256eb4-eec2-4d39-8420-5403098c7cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"
Sep 17 16:43:08.297548 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.297383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70256eb4-eec2-4d39-8420-5403098c7cbf-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-5lwqk\" (UID: \"70256eb4-eec2-4d39-8420-5403098c7cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"
Sep 17 16:43:08.298238 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.298197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70256eb4-eec2-4d39-8420-5403098c7cbf-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-5lwqk\" (UID: \"70256eb4-eec2-4d39-8420-5403098c7cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"
Sep 17 16:43:08.300301 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.300279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70256eb4-eec2-4d39-8420-5403098c7cbf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-5lwqk\" (UID: \"70256eb4-eec2-4d39-8420-5403098c7cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"
Sep 17 16:43:08.300418 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.300305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/70256eb4-eec2-4d39-8420-5403098c7cbf-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-5lwqk\" (UID: \"70256eb4-eec2-4d39-8420-5403098c7cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"
Sep 17 16:43:08.308337 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.308313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqdx5\" (UniqueName: \"kubernetes.io/projected/70256eb4-eec2-4d39-8420-5403098c7cbf-kube-api-access-kqdx5\") pod \"prometheus-operator-7bd5d56d6-5lwqk\" (UID: \"70256eb4-eec2-4d39-8420-5403098c7cbf\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"
Sep 17 16:43:08.356704 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.356680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"
Sep 17 16:43:08.499540 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.499506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs\") pod \"network-metrics-daemon-24wdv\" (UID: \"d306fc98-5114-4b68-a4f7-bfaaf0e419d1\") " pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:43:08.502291 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.502266    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 16:43:08.513420 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.513392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d306fc98-5114-4b68-a4f7-bfaaf0e419d1-metrics-certs\") pod \"network-metrics-daemon-24wdv\" (UID: \"d306fc98-5114-4b68-a4f7-bfaaf0e419d1\") " pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:43:08.601015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.600906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:43:08.604075 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.604051    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 16:43:08.614725 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.614694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/667acd31-9d7e-4010-bbb3-f18b17232f48-original-pull-secret\") pod \"global-pull-secret-syncer-lgk7x\" (UID: \"667acd31-9d7e-4010-bbb3-f18b17232f48\") " pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:43:08.702346 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.702313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmc9z\" (UniqueName: \"kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z\") pod \"network-check-target-qz854\" (UID: \"469c498c-7671-4d9b-ae5d-ee75d043b4d4\") " pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:43:08.705397 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.705369    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 16:43:08.714882 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.714860    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 16:43:08.725868 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.725849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmc9z\" (UniqueName: \"kubernetes.io/projected/469c498c-7671-4d9b-ae5d-ee75d043b4d4-kube-api-access-kmc9z\") pod \"network-check-target-qz854\" (UID: \"469c498c-7671-4d9b-ae5d-ee75d043b4d4\") " pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:43:08.743760 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.743732    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgk7x"
Sep 17 16:43:08.753951 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.753930    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k782z\""
Sep 17 16:43:08.759219 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.759202    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hqd8r\""
Sep 17 16:43:08.762024 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.762006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24wdv"
Sep 17 16:43:08.767431 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:08.767368    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:43:11.208398 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:11.208333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z" event={"ID":"582fad66-8507-413f-bfcc-5b6917d21e64","Type":"ContainerStarted","Data":"4067eaf5611f1812c3e433160fdf9b327d972a0aacb030684d8359a409fdcd13"}
Sep 17 16:43:11.208878 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:11.208684    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"
Sep 17 16:43:11.210101 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:11.210075    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z"
Sep 17 16:43:11.228113 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:11.228036    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65544fff6b-gs55z" podStartSLOduration=1.308760068 podStartE2EDuration="9.228020414s" totalImagesPullingTime="7.919260346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:11.226387372 +0000 UTC m=+70.649325236" watchObservedRunningTime="2026-09-17 16:43:11.228020414 +0000 UTC m=+70.650958278"
Sep 17 16:43:11.277214 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:11.277181    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lgk7x"]
Sep 17 16:43:11.278268 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:11.278248    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-24wdv"]
Sep 17 16:43:11.282621 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:11.282601    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk"]
Sep 17 16:43:11.282852 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:11.282831    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qz854"]
Sep 17 16:43:11.317164 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:43:11.317131    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667acd31_9d7e_4010_bbb3_f18b17232f48.slice/crio-bf71e373d05522cf7268950cf7b5f78f84d3ff605a1d53c6f7e3a0b2af0401c5 WatchSource:0}: Error finding container bf71e373d05522cf7268950cf7b5f78f84d3ff605a1d53c6f7e3a0b2af0401c5: Status 404 returned error can't find the container with id bf71e373d05522cf7268950cf7b5f78f84d3ff605a1d53c6f7e3a0b2af0401c5
Sep 17 16:43:11.318192 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:43:11.318163    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd306fc98_5114_4b68_a4f7_bfaaf0e419d1.slice/crio-0519ec0501e48f143420882366cc5ab58204aefe266e4c140b9c22353e0d363d WatchSource:0}: Error finding container 0519ec0501e48f143420882366cc5ab58204aefe266e4c140b9c22353e0d363d: Status 404 returned error can't find the container with id 0519ec0501e48f143420882366cc5ab58204aefe266e4c140b9c22353e0d363d
Sep 17 16:43:11.319471 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:43:11.319444    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70256eb4_eec2_4d39_8420_5403098c7cbf.slice/crio-26db3df11b8918e631c069958288b1eab62d26e536aa0396322a6c38cf721837 WatchSource:0}: Error finding container 26db3df11b8918e631c069958288b1eab62d26e536aa0396322a6c38cf721837: Status 404 returned error can't find the container with id 26db3df11b8918e631c069958288b1eab62d26e536aa0396322a6c38cf721837
Sep 17 16:43:11.321203 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:43:11.321182    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469c498c_7671_4d9b_ae5d_ee75d043b4d4.slice/crio-542cec1fdb877fe6525161fc6764bea6ffbf70a0915378d3286e3ab1c0007204 WatchSource:0}: Error finding container 542cec1fdb877fe6525161fc6764bea6ffbf70a0915378d3286e3ab1c0007204: Status 404 returned error can't find the container with id 542cec1fdb877fe6525161fc6764bea6ffbf70a0915378d3286e3ab1c0007204
Sep 17 16:43:12.220995 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:12.220937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-24wdv" event={"ID":"d306fc98-5114-4b68-a4f7-bfaaf0e419d1","Type":"ContainerStarted","Data":"0519ec0501e48f143420882366cc5ab58204aefe266e4c140b9c22353e0d363d"}
Sep 17 16:43:12.224139 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:12.224112    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qz854" event={"ID":"469c498c-7671-4d9b-ae5d-ee75d043b4d4","Type":"ContainerStarted","Data":"542cec1fdb877fe6525161fc6764bea6ffbf70a0915378d3286e3ab1c0007204"}
Sep 17 16:43:12.228037 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:12.227303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695b7bc4c4-r96j8" event={"ID":"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b","Type":"ContainerStarted","Data":"38995bac0c2cbb79de64bb70672d1683ccb7665079222e623cbdeda12e5be188"}
Sep 17 16:43:12.230281 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:12.230214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lgk7x" event={"ID":"667acd31-9d7e-4010-bbb3-f18b17232f48","Type":"ContainerStarted","Data":"bf71e373d05522cf7268950cf7b5f78f84d3ff605a1d53c6f7e3a0b2af0401c5"}
Sep 17 16:43:12.233784 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:12.233725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk" event={"ID":"70256eb4-eec2-4d39-8420-5403098c7cbf","Type":"ContainerStarted","Data":"26db3df11b8918e631c069958288b1eab62d26e536aa0396322a6c38cf721837"}
Sep 17 16:43:12.926325 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:12.925790    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:12.926325 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:12.925920    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:12.932117 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:12.932061    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:12.971801 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:12.971751    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-695b7bc4c4-r96j8" podStartSLOduration=2.835359845 podStartE2EDuration="10.971731019s" totalImagesPullingTime="8.136371174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:12.253097775 +0000 UTC m=+71.676035662" watchObservedRunningTime="2026-09-17 16:43:12.971731019 +0000 UTC m=+72.394668885"
Sep 17 16:43:13.244041 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:13.244012    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:14.246126 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:14.246079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk" event={"ID":"70256eb4-eec2-4d39-8420-5403098c7cbf","Type":"ContainerStarted","Data":"e954520d229c6c9b9cc5aaf1042431e14e63bcea8bc93df6fbb012aa772402ca"}
Sep 17 16:43:14.246591 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:14.246188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk" event={"ID":"70256eb4-eec2-4d39-8420-5403098c7cbf","Type":"ContainerStarted","Data":"b2974e82f157151548584ea64b93d1410b4e7dedd9654947d74ac36d5e15e025"}
Sep 17 16:43:14.249584 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:14.249546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-24wdv" event={"ID":"d306fc98-5114-4b68-a4f7-bfaaf0e419d1","Type":"ContainerStarted","Data":"72a5be1d5dddfb18a7d3438c8b29a214c25424e250c66cadd1e8d934c2ba61a1"}
Sep 17 16:43:14.249734 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:14.249605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-24wdv" event={"ID":"d306fc98-5114-4b68-a4f7-bfaaf0e419d1","Type":"ContainerStarted","Data":"cc7976d566bb547d7eda81be5fb82d16412652f17d4804d326f107a898cc10dc"}
Sep 17 16:43:14.266988 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:14.266516    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-5lwqk" podStartSLOduration=4.193159015 podStartE2EDuration="6.266498005s" totalImagesPullingTime="2.07333899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:14.265662775 +0000 UTC m=+73.688600665" watchObservedRunningTime="2026-09-17 16:43:14.266498005 +0000 UTC m=+73.689435868"
Sep 17 16:43:14.286579 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:14.286406    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-24wdv" podStartSLOduration=70.212497995 podStartE2EDuration="1m12.286387003s" totalImagesPullingTime="2.073889008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:14.285485705 +0000 UTC m=+73.708423570" watchObservedRunningTime="2026-09-17 16:43:14.286387003 +0000 UTC m=+73.709324869"
Sep 17 16:43:16.480337 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.480283    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vh4k5"]
Sep 17 16:43:16.482925 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.482900    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-45n6z"]
Sep 17 16:43:16.483103 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.483074    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.485265 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.485233    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.485506 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.485485    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bwswv\""
Sep 17 16:43:16.485876 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.485826    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 16:43:16.486015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.485985    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 16:43:16.486239 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.486222    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 16:43:16.487816 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.487793    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 16:43:16.488019 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.487801    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 16:43:16.488139 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.488114    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 16:43:16.488218 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.488143    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xmp5j\""
Sep 17 16:43:16.495357 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.495340    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-45n6z"]
Sep 17 16:43:16.574716 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.574691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.574861 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.574728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2zn4\" (UniqueName: \"kubernetes.io/projected/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-api-access-q2zn4\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.574861 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.574758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-accelerators-collector-config\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.574947 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.574871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-tls\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.574947 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.574918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgctp\" (UniqueName: \"kubernetes.io/projected/2a27dbb7-6616-42f2-b562-b7aebfb49adb-kube-api-access-kgctp\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.575056 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.574944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a27dbb7-6616-42f2-b562-b7aebfb49adb-root\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.575056 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.575003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.575056 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.575045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.575170 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.575074    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-wtmp\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.575170 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.575098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a27dbb7-6616-42f2-b562-b7aebfb49adb-metrics-client-ca\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.575170 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.575119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a27dbb7-6616-42f2-b562-b7aebfb49adb-sys\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.575170 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.575152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.575305 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.575178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-textfile\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.575305 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.575204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.575305 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.575230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.675789 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.675758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.675804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-textfile\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.675832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.675867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.675909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.675937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2zn4\" (UniqueName: \"kubernetes.io/projected/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-api-access-q2zn4\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.675993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-accelerators-collector-config\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.676031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-tls\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.676059    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgctp\" (UniqueName: \"kubernetes.io/projected/2a27dbb7-6616-42f2-b562-b7aebfb49adb-kube-api-access-kgctp\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.676080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a27dbb7-6616-42f2-b562-b7aebfb49adb-root\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:43:16.676103    2581 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.676116    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.676149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:43:16.676171    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-state-metrics-tls podName:72b72d14-1c90-49a1-adfe-8d9d62e9c7f4 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:17.176148142 +0000 UTC m=+76.599085997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-45n6z" (UID: "72b72d14-1c90-49a1-adfe-8d9d62e9c7f4") : secret "kube-state-metrics-tls" not found
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.676203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-wtmp\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.676243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a27dbb7-6616-42f2-b562-b7aebfb49adb-metrics-client-ca\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.677849 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.676270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a27dbb7-6616-42f2-b562-b7aebfb49adb-sys\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.678733 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.676319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.678733 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.676347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a27dbb7-6616-42f2-b562-b7aebfb49adb-sys\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.678733 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.676575    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-textfile\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.678733 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.676695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-wtmp\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.678733 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.676957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.678733 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.676988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a27dbb7-6616-42f2-b562-b7aebfb49adb-root\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.678733 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.677505    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a27dbb7-6616-42f2-b562-b7aebfb49adb-metrics-client-ca\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.678733 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.677798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.678733 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.678299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-accelerators-collector-config\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.680456 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.680433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.680843 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.680826    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.681203 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.681188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a27dbb7-6616-42f2-b562-b7aebfb49adb-node-exporter-tls\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.688193 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.688170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgctp\" (UniqueName: \"kubernetes.io/projected/2a27dbb7-6616-42f2-b562-b7aebfb49adb-kube-api-access-kgctp\") pod \"node-exporter-vh4k5\" (UID: \"2a27dbb7-6616-42f2-b562-b7aebfb49adb\") " pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:16.689571 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.689548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2zn4\" (UniqueName: \"kubernetes.io/projected/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-api-access-q2zn4\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:16.796243 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:16.796064    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vh4k5"
Sep 17 16:43:17.005872 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.005839    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-695b7bc4c4-r96j8"]
Sep 17 16:43:17.180569 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.180490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:17.183005 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.182979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/72b72d14-1c90-49a1-adfe-8d9d62e9c7f4-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-45n6z\" (UID: \"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:17.402979 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.402945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z"
Sep 17 16:43:17.506136 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.506101    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 16:43:17.509096 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.509034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.512825 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.512799    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 16:43:17.514389 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.514017    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 16:43:17.514389 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.514087    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 16:43:17.514389 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.514104    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 16:43:17.514389 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.514264    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 16:43:17.514389 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.514341    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lnpfw\""
Sep 17 16:43:17.514714 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.514466    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 16:43:17.514714 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.514545    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 16:43:17.516316 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.516296    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 16:43:17.517906 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.517889    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 16:43:17.541554 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.541533    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 16:43:17.684774 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.684741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.684934 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.684797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-web-config\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.684934 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.684880    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.685055 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.684937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.685055 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.684987    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.685055 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.685017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.685176 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.685086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.685176 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.685117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-config-out\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.685251 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.685173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.685251 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.685198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.685251 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.685215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.685251 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.685232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.685251 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.685250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsrd2\" (UniqueName: \"kubernetes.io/projected/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-kube-api-access-xsrd2\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.786255 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.786178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.786255 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.786221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-config-out\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.786472 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.786260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.786472 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.786284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.786472 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.786309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.786472 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.786332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.786472 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.786356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsrd2\" (UniqueName: \"kubernetes.io/projected/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-kube-api-access-xsrd2\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.786472 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.786380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.786472 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.786410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-web-config\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.786472 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:43:17.786430    2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 17 16:43:17.786472 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.786447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.786892 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.786481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.786892 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:43:17.786498    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-main-tls podName:3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:18.286475235 +0000 UTC m=+77.709413093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6") : secret "alertmanager-main-tls" not found
Sep 17 16:43:17.786892 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.786540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.786892 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.786573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.787924 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.787631    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.788208 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.788180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.788488 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.788466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.789428 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.789383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.789908 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.789843    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-web-config\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.790255 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.790214    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-config-out\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.791080 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.791038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.791652 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.791619    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.791863 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.791823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.792090 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.792069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.793459 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.793436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:17.796591 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:17.796569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsrd2\" (UniqueName: \"kubernetes.io/projected/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-kube-api-access-xsrd2\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:18.291657 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:18.291629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:18.294228 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:18.294204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:18.421550 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:18.421516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 16:43:21.252466 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.252423    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq"]
Sep 17 16:43:21.321534 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.321494    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq"]
Sep 17 16:43:21.321692 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.321613    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq"
Sep 17 16:43:21.324077 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.324055    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 16:43:21.324201 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.324125    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-f75m5\""
Sep 17 16:43:21.418410 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.418379    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-pdfvq\" (UID: \"9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq"
Sep 17 16:43:21.418546 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.418456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qhbw\" (UniqueName: \"kubernetes.io/projected/9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5-kube-api-access-7qhbw\") pod \"monitoring-plugin-5984459ff8-pdfvq\" (UID: \"9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq"
Sep 17 16:43:21.519235 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.519169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-pdfvq\" (UID: \"9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq"
Sep 17 16:43:21.519359 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.519236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qhbw\" (UniqueName: \"kubernetes.io/projected/9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5-kube-api-access-7qhbw\") pod \"monitoring-plugin-5984459ff8-pdfvq\" (UID: \"9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq"
Sep 17 16:43:21.522075 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.522046    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-pdfvq\" (UID: \"9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq"
Sep 17 16:43:21.531426 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.531405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qhbw\" (UniqueName: \"kubernetes.io/projected/9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5-kube-api-access-7qhbw\") pod \"monitoring-plugin-5984459ff8-pdfvq\" (UID: \"9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq"
Sep 17 16:43:21.634030 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.633998    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq"
Sep 17 16:43:21.775598 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.775525    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"]
Sep 17 16:43:21.792738 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.792717    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:21.795724 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.795702    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 16:43:21.795999 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.795945    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 16:43:21.796131 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.796024    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 16:43:21.796131 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.796024    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 16:43:21.796361 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.796328    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9lgb5\""
Sep 17 16:43:21.796477 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.796371    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 16:43:21.812690 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.812666    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"]
Sep 17 16:43:21.816171 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.816150    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 16:43:21.923234 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.923208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e7e2b070-60a7-4018-8457-1676ebd17592-federate-client-tls\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:21.923380 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.923244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7e2b070-60a7-4018-8457-1676ebd17592-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:21.923380 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.923278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cqcj\" (UniqueName: \"kubernetes.io/projected/e7e2b070-60a7-4018-8457-1676ebd17592-kube-api-access-2cqcj\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:21.923380 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.923359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e2b070-60a7-4018-8457-1676ebd17592-serving-certs-ca-bundle\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:21.923520 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.923396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e2b070-60a7-4018-8457-1676ebd17592-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:21.923520 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.923428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e7e2b070-60a7-4018-8457-1676ebd17592-secret-telemeter-client\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:21.923520 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.923497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e7e2b070-60a7-4018-8457-1676ebd17592-telemeter-client-tls\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:21.923623 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:21.923526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7e2b070-60a7-4018-8457-1676ebd17592-metrics-client-ca\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.024144 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.024108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e7e2b070-60a7-4018-8457-1676ebd17592-federate-client-tls\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.024307 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.024159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7e2b070-60a7-4018-8457-1676ebd17592-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.024307 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.024209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cqcj\" (UniqueName: \"kubernetes.io/projected/e7e2b070-60a7-4018-8457-1676ebd17592-kube-api-access-2cqcj\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.024307 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.024256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e2b070-60a7-4018-8457-1676ebd17592-serving-certs-ca-bundle\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.024307 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.024288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e2b070-60a7-4018-8457-1676ebd17592-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.024522 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.024321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e7e2b070-60a7-4018-8457-1676ebd17592-secret-telemeter-client\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.024522 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.024366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e7e2b070-60a7-4018-8457-1676ebd17592-telemeter-client-tls\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.024522 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.024391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7e2b070-60a7-4018-8457-1676ebd17592-metrics-client-ca\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.025125 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.025097    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e2b070-60a7-4018-8457-1676ebd17592-serving-certs-ca-bundle\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.025305 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.025282    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7e2b070-60a7-4018-8457-1676ebd17592-metrics-client-ca\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.025593 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.025566    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e2b070-60a7-4018-8457-1676ebd17592-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.027631 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.027555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e7e2b070-60a7-4018-8457-1676ebd17592-telemeter-client-tls\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.027887 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.027859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e7e2b070-60a7-4018-8457-1676ebd17592-federate-client-tls\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.027997 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.027915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e7e2b070-60a7-4018-8457-1676ebd17592-secret-telemeter-client\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.028377 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.028350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7e2b070-60a7-4018-8457-1676ebd17592-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.034345 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.034322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cqcj\" (UniqueName: \"kubernetes.io/projected/e7e2b070-60a7-4018-8457-1676ebd17592-kube-api-access-2cqcj\") pod \"telemeter-client-6c9c9875f6-9pxqv\" (UID: \"e7e2b070-60a7-4018-8457-1676ebd17592\") " pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:22.104180 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:22.104159    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"
Sep 17 16:43:24.544755 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:43:24.544635    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a27dbb7_6616_42f2_b562_b7aebfb49adb.slice/crio-24ac720bf118cea45434cf25411c1251a68781b2e1c91ec17742d177b1ade7da WatchSource:0}: Error finding container 24ac720bf118cea45434cf25411c1251a68781b2e1c91ec17742d177b1ade7da: Status 404 returned error can't find the container with id 24ac720bf118cea45434cf25411c1251a68781b2e1c91ec17742d177b1ade7da
Sep 17 16:43:24.841226 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:43:24.839427    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0d9dc0_af8b_4ab5_8d8d_c020ee1209b5.slice/crio-81554b2552b09d36db9f5f2b8c6405e307fdfaaddb605b9f95d3bc7f2ff60bf1 WatchSource:0}: Error finding container 81554b2552b09d36db9f5f2b8c6405e307fdfaaddb605b9f95d3bc7f2ff60bf1: Status 404 returned error can't find the container with id 81554b2552b09d36db9f5f2b8c6405e307fdfaaddb605b9f95d3bc7f2ff60bf1
Sep 17 16:43:24.841226 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:24.840441    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq"]
Sep 17 16:43:24.843034 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:24.842601    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-45n6z"]
Sep 17 16:43:24.843320 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:43:24.843142    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b72d14_1c90_49a1_adfe_8d9d62e9c7f4.slice/crio-9a30419045b3c56c47f1f508b9ce6421ed02da04735304b31ad28c03953c4d6c WatchSource:0}: Error finding container 9a30419045b3c56c47f1f508b9ce6421ed02da04735304b31ad28c03953c4d6c: Status 404 returned error can't find the container with id 9a30419045b3c56c47f1f508b9ce6421ed02da04735304b31ad28c03953c4d6c
Sep 17 16:43:24.849891 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:24.849866    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv"]
Sep 17 16:43:24.850434 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:24.850416    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 16:43:24.851696 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:43:24.851660    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3525b5_bf6e_4ecd_bb27_185fd0f6fec6.slice/crio-3161316de61eedd92d40c0bf63267da856136f10803d4614ef6cd19a91f8f3a8 WatchSource:0}: Error finding container 3161316de61eedd92d40c0bf63267da856136f10803d4614ef6cd19a91f8f3a8: Status 404 returned error can't find the container with id 3161316de61eedd92d40c0bf63267da856136f10803d4614ef6cd19a91f8f3a8
Sep 17 16:43:24.854148 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:43:24.854129    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e2b070_60a7_4018_8457_1676ebd17592.slice/crio-db492dcc5345f9ac562fcb83f2b64954b4b6bdca9b31a9d808f7b4a9bab64b53 WatchSource:0}: Error finding container db492dcc5345f9ac562fcb83f2b64954b4b6bdca9b31a9d808f7b4a9bab64b53: Status 404 returned error can't find the container with id db492dcc5345f9ac562fcb83f2b64954b4b6bdca9b31a9d808f7b4a9bab64b53
Sep 17 16:43:25.295469 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:25.295430    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z" event={"ID":"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4","Type":"ContainerStarted","Data":"9a30419045b3c56c47f1f508b9ce6421ed02da04735304b31ad28c03953c4d6c"}
Sep 17 16:43:25.297389 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:25.297317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh4k5" event={"ID":"2a27dbb7-6616-42f2-b562-b7aebfb49adb","Type":"ContainerStarted","Data":"24ac720bf118cea45434cf25411c1251a68781b2e1c91ec17742d177b1ade7da"}
Sep 17 16:43:25.300269 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:25.299750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qz854" event={"ID":"469c498c-7671-4d9b-ae5d-ee75d043b4d4","Type":"ContainerStarted","Data":"0c27c11fd97e702ec04ca359e28c22d594139918f2933007ec113bf188b0c67f"}
Sep 17 16:43:25.300269 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:25.300212    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:43:25.308933 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:25.308907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6","Type":"ContainerStarted","Data":"3161316de61eedd92d40c0bf63267da856136f10803d4614ef6cd19a91f8f3a8"}
Sep 17 16:43:25.313265 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:25.313113    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv" event={"ID":"e7e2b070-60a7-4018-8457-1676ebd17592","Type":"ContainerStarted","Data":"db492dcc5345f9ac562fcb83f2b64954b4b6bdca9b31a9d808f7b4a9bab64b53"}
Sep 17 16:43:25.315878 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:25.315736    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lgk7x" event={"ID":"667acd31-9d7e-4010-bbb3-f18b17232f48","Type":"ContainerStarted","Data":"8e4bcc3000c241b28e1c925d5e59aa43d7ffa46bed9661ad34780d9914f83780"}
Sep 17 16:43:25.319759 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:25.319562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-6h4np" event={"ID":"4eb30ced-c505-47e5-aca7-abbe51e37473","Type":"ContainerStarted","Data":"10c691638f104e480532aad9416c735ebc4707bb9b9e091f0f49978690ccd0f9"}
Sep 17 16:43:25.320427 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:25.320337    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-6h4np"
Sep 17 16:43:25.323689 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:25.323643    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qz854" podStartSLOduration=70.114850963 podStartE2EDuration="1m23.323629959s" totalImagesPullingTime="13.208778996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:25.321160526 +0000 UTC m=+84.744098390" watchObservedRunningTime="2026-09-17 16:43:25.323629959 +0000 UTC m=+84.746567822"
Sep 17 16:43:25.324534 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:25.324327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq" event={"ID":"9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5","Type":"ContainerStarted","Data":"81554b2552b09d36db9f5f2b8c6405e307fdfaaddb605b9f95d3bc7f2ff60bf1"}
Sep 17 16:43:25.335832 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:25.335792    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-6h4np"
Sep 17 16:43:25.345369 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:25.345234    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lgk7x" podStartSLOduration=69.136068829 podStartE2EDuration="1m22.345216418s" totalImagesPullingTime="13.209147589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:25.343841148 +0000 UTC m=+84.766779012" watchObservedRunningTime="2026-09-17 16:43:25.345216418 +0000 UTC m=+84.768154282"
Sep 17 16:43:25.410543 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:25.409012    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-6h4np" podStartSLOduration=2.01199815 podStartE2EDuration="23.40898849s" totalImagesPullingTime="21.39699034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:25.367448039 +0000 UTC m=+84.790386131" watchObservedRunningTime="2026-09-17 16:43:25.40898849 +0000 UTC m=+84.831926370"
Sep 17 16:43:26.194484 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:26.194436    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-555b8df8cf-7hc5c"
Sep 17 16:43:30.354219 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.354177    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6" containerID="d8555dad06984b2aaf23bd8717f9e3b9fc3ec021d90d7fdbb62482d91896e2ac" exitCode=0
Sep 17 16:43:30.354696 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.354270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6","Type":"ContainerDied","Data":"d8555dad06984b2aaf23bd8717f9e3b9fc3ec021d90d7fdbb62482d91896e2ac"}
Sep 17 16:43:30.357517 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.357210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv" event={"ID":"e7e2b070-60a7-4018-8457-1676ebd17592","Type":"ContainerStarted","Data":"585c8e66fc4834a818a2990ccac8baa89452b59da827d726abcfbed13719c203"}
Sep 17 16:43:30.357626 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.357565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv" event={"ID":"e7e2b070-60a7-4018-8457-1676ebd17592","Type":"ContainerStarted","Data":"39f8bb6926367e2d3263b86ffba16beb680979ab76f3c92102c0447670930557"}
Sep 17 16:43:30.357626 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.357602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv" event={"ID":"e7e2b070-60a7-4018-8457-1676ebd17592","Type":"ContainerStarted","Data":"e2337fd0c7fd5342b85159b0fff34a311833691198246af966ac9a9c0974c7c4"}
Sep 17 16:43:30.359338 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.359311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq" event={"ID":"9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5","Type":"ContainerStarted","Data":"bb4e9301ebefdea65a1b8a53c4922ddc0e49fb7b1268f5094f6f9d3c6eafcb1e"}
Sep 17 16:43:30.359579 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.359529    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq"
Sep 17 16:43:30.361936 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.361907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z" event={"ID":"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4","Type":"ContainerStarted","Data":"b838969b3b6b765090f407ab98a6c36ad137ff4b8a03592b18ee2e59157252af"}
Sep 17 16:43:30.362054 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.361988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z" event={"ID":"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4","Type":"ContainerStarted","Data":"1d2471eb8a3895873a70fca6faacc3dcf4124c9be1278d053164633bfcbdb76d"}
Sep 17 16:43:30.362054 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.362029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z" event={"ID":"72b72d14-1c90-49a1-adfe-8d9d62e9c7f4","Type":"ContainerStarted","Data":"39e9fe79e70d2189cdaea3cf48b389d99b75d19543954d54fe6a722de881377d"}
Sep 17 16:43:30.364278 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.364252    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a27dbb7-6616-42f2-b562-b7aebfb49adb" containerID="ed7cb9105c529ca8338bb0a421aa6f8358b55501e62e8ff2eb80e979852910ff" exitCode=0
Sep 17 16:43:30.364380 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.364307    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh4k5" event={"ID":"2a27dbb7-6616-42f2-b562-b7aebfb49adb","Type":"ContainerDied","Data":"ed7cb9105c529ca8338bb0a421aa6f8358b55501e62e8ff2eb80e979852910ff"}
Sep 17 16:43:30.365539 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.365515    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq"
Sep 17 16:43:30.439524 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.439473    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-pdfvq" podStartSLOduration=4.907588866 podStartE2EDuration="9.439455216s" totalImagesPullingTime="4.53186635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:30.439370214 +0000 UTC m=+89.862308077" watchObservedRunningTime="2026-09-17 16:43:30.439455216 +0000 UTC m=+89.862393095"
Sep 17 16:43:30.469145 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.468958    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6c9c9875f6-9pxqv" podStartSLOduration=4.617546047 podStartE2EDuration="9.468940783s" totalImagesPullingTime="4.851394736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:30.467225716 +0000 UTC m=+89.890163580" watchObservedRunningTime="2026-09-17 16:43:30.468940783 +0000 UTC m=+89.891878648"
Sep 17 16:43:30.529640 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:30.529587    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-45n6z" podStartSLOduration=9.997414919 podStartE2EDuration="14.529566991s" totalImagesPullingTime="4.532152072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:30.528119206 +0000 UTC m=+89.951057070" watchObservedRunningTime="2026-09-17 16:43:30.529566991 +0000 UTC m=+89.952504855"
Sep 17 16:43:31.379800 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:31.379745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh4k5" event={"ID":"2a27dbb7-6616-42f2-b562-b7aebfb49adb","Type":"ContainerStarted","Data":"15980ea97837496454d1ca9efbc18b3f22e3aa323abf15917c88999bed310289"}
Sep 17 16:43:31.380351 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:31.379824    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vh4k5" event={"ID":"2a27dbb7-6616-42f2-b562-b7aebfb49adb","Type":"ContainerStarted","Data":"1a7b9bdd99c3b8d8def695a831daa266a71058361cce19b1a57e5854d3b711ff"}
Sep 17 16:43:31.410394 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:31.410333    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vh4k5" podStartSLOduration=10.4621805 podStartE2EDuration="15.410313371s" totalImagesPullingTime="4.824908269s" totalInitContainerRuntime="123.224602ms" isStatefulPod=true podCreationTimestamp="2026-09-17 16:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:31.407736247 +0000 UTC m=+90.830674111" watchObservedRunningTime="2026-09-17 16:43:31.410313371 +0000 UTC m=+90.833251237"
Sep 17 16:43:33.393466 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:33.393391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6","Type":"ContainerStarted","Data":"8518e84ca34903b420728ca7a4b7ee968a6aa72ee83e0a2ef9935a1c3ec73513"}
Sep 17 16:43:33.393796 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:33.393480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6","Type":"ContainerStarted","Data":"4715930f51ee5170013426c471272865f97c757e0891400a9ad43937d3e9e416"}
Sep 17 16:43:33.393796 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:33.393509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6","Type":"ContainerStarted","Data":"425e147e9412e8e196125cf1ca9f8a709a8d124cf692f327d4b862d9da5aca90"}
Sep 17 16:43:33.393796 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:33.393534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6","Type":"ContainerStarted","Data":"255b041e75d2322df81104215e41019f1d462be10a366747c0a67bdb074144de"}
Sep 17 16:43:33.393796 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:33.393556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6","Type":"ContainerStarted","Data":"2837151776c3eb4d30a02f240d28a2ebacb312569a3b519ebe3c12e12a6f6c5a"}
Sep 17 16:43:35.405569 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:35.405531    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6","Type":"ContainerStarted","Data":"6c009d33d5a88d9f343263edec10a6908e277005fc3bc15951c641d3a95a4eeb"}
Sep 17 16:43:35.445384 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:35.445336    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=10.628765082 podStartE2EDuration="18.445316152s" totalImagesPullingTime="7.798565651s" totalInitContainerRuntime="17.985419ms" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:35.443530479 +0000 UTC m=+94.866468343" watchObservedRunningTime="2026-09-17 16:43:35.445316152 +0000 UTC m=+94.868254015"
Sep 17 16:43:42.027746 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.027683    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-695b7bc4c4-r96j8" podUID="1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b" containerName="console" containerID="cri-o://38995bac0c2cbb79de64bb70672d1683ccb7665079222e623cbdeda12e5be188" gracePeriod=15
Sep 17 16:43:42.278780 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.278729    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:42.292201 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.292174    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-oauth-config\") pod \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") "
Sep 17 16:43:42.292296 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.292206    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-serving-cert\") pod \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") "
Sep 17 16:43:42.292296 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.292246    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg5fg\" (UniqueName: \"kubernetes.io/projected/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-kube-api-access-tg5fg\") pod \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") "
Sep 17 16:43:42.292442 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.292419    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-config\") pod \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") "
Sep 17 16:43:42.292497 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.292466    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-oauth-serving-cert\") pod \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") "
Sep 17 16:43:42.292556 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.292501    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-service-ca\") pod \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\" (UID: \"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b\") "
Sep 17 16:43:42.292862 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.292834    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-config" (OuterVolumeSpecName: "console-config") pod "1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b" (UID: "1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:43:42.292952 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.292875    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b" (UID: "1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:43:42.293067 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.293044    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-service-ca" (OuterVolumeSpecName: "service-ca") pod "1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b" (UID: "1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:43:42.294552 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.294527    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b" (UID: "1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:43:42.294832 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.294812    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b" (UID: "1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:43:42.294885 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.294859    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-kube-api-access-tg5fg" (OuterVolumeSpecName: "kube-api-access-tg5fg") pod "1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b" (UID: "1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b"). InnerVolumeSpecName "kube-api-access-tg5fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:43:42.393585 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.393562    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-oauth-config\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:43:42.393585 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.393583    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-serving-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:43:42.393727 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.393593    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg5fg\" (UniqueName: \"kubernetes.io/projected/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-kube-api-access-tg5fg\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:43:42.393727 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.393602    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-console-config\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:43:42.393727 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.393610    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-oauth-serving-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:43:42.393727 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.393618    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b-service-ca\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:43:42.437553 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.437523    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b" containerID="38995bac0c2cbb79de64bb70672d1683ccb7665079222e623cbdeda12e5be188" exitCode=0
Sep 17 16:43:42.437642 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.437569    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695b7bc4c4-r96j8"
Sep 17 16:43:42.437642 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.437607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695b7bc4c4-r96j8" event={"ID":"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b","Type":"ContainerDied","Data":"38995bac0c2cbb79de64bb70672d1683ccb7665079222e623cbdeda12e5be188"}
Sep 17 16:43:42.437719 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.437679    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695b7bc4c4-r96j8" event={"ID":"1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b","Type":"ContainerDied","Data":"eb9f9c69cd80d6372dea02e6f8ad85e5adc42e899bc39fc7ad01a86968610a06"}
Sep 17 16:43:42.437719 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.437709    2581 scope.go:117] "RemoveContainer" containerID="38995bac0c2cbb79de64bb70672d1683ccb7665079222e623cbdeda12e5be188"
Sep 17 16:43:42.454316 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.454299    2581 scope.go:117] "RemoveContainer" containerID="38995bac0c2cbb79de64bb70672d1683ccb7665079222e623cbdeda12e5be188"
Sep 17 16:43:42.454588 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:43:42.454568    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38995bac0c2cbb79de64bb70672d1683ccb7665079222e623cbdeda12e5be188\": container with ID starting with 38995bac0c2cbb79de64bb70672d1683ccb7665079222e623cbdeda12e5be188 not found: ID does not exist" containerID="38995bac0c2cbb79de64bb70672d1683ccb7665079222e623cbdeda12e5be188"
Sep 17 16:43:42.454662 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.454600    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38995bac0c2cbb79de64bb70672d1683ccb7665079222e623cbdeda12e5be188"} err="failed to get container status \"38995bac0c2cbb79de64bb70672d1683ccb7665079222e623cbdeda12e5be188\": rpc error: code = NotFound desc = could not find container \"38995bac0c2cbb79de64bb70672d1683ccb7665079222e623cbdeda12e5be188\": container with ID starting with 38995bac0c2cbb79de64bb70672d1683ccb7665079222e623cbdeda12e5be188 not found: ID does not exist"
Sep 17 16:43:42.459250 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.459230    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-695b7bc4c4-r96j8"]
Sep 17 16:43:42.464438 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.464419    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-695b7bc4c4-r96j8"]
Sep 17 16:43:42.834474 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:42.834452    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b" path="/var/lib/kubelet/pods/1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b/volumes"
Sep 17 16:43:56.337317 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:43:56.337286    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qz854"
Sep 17 16:44:40.247435 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.247400    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-744b6569b4-7bpw8"]
Sep 17 16:44:40.247829 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.247698    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b" containerName="console"
Sep 17 16:44:40.247829 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.247711    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b" containerName="console"
Sep 17 16:44:40.247829 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.247763    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e6aa0fa-a25e-4a5c-a200-cb2e1b146b0b" containerName="console"
Sep 17 16:44:40.250925 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.250895    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.254063 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.254035    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 16:44:40.254233 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.254035    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 16:44:40.254233 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.254114    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 16:44:40.254233 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.254038    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 16:44:40.254233 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.254049    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7s58x\""
Sep 17 16:44:40.254507 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.254492    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 16:44:40.258550 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.258525    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 16:44:40.267200 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.267171    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-744b6569b4-7bpw8"]
Sep 17 16:44:40.320377 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.320357    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-oauth-config\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.320478 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.320400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-oauth-serving-cert\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.320478 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.320421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-config\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.320554 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.320490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-serving-cert\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.320554 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.320522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-service-ca\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.320634 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.320558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-trusted-ca-bundle\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.320634 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.320586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pjdz\" (UniqueName: \"kubernetes.io/projected/3eb3d00b-a6e9-4172-b79f-108213bc9d48-kube-api-access-5pjdz\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.420998 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.420956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-serving-cert\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.421127 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.421002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-service-ca\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.421127 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.421028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-trusted-ca-bundle\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.421127 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.421045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjdz\" (UniqueName: \"kubernetes.io/projected/3eb3d00b-a6e9-4172-b79f-108213bc9d48-kube-api-access-5pjdz\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.421127 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.421066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-oauth-config\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.421127 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.421100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-oauth-serving-cert\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.421127 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.421119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-config\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.421843 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.421814    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-service-ca\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.421843 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.421841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-oauth-serving-cert\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.422054 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.421841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-config\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.422054 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.421954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-trusted-ca-bundle\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.423479 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.423462    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-oauth-config\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.423606 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.423588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-serving-cert\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.432451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.432427    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjdz\" (UniqueName: \"kubernetes.io/projected/3eb3d00b-a6e9-4172-b79f-108213bc9d48-kube-api-access-5pjdz\") pod \"console-744b6569b4-7bpw8\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") " pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.562223 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.562168    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:40.837651 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:40.837564    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-744b6569b4-7bpw8"]
Sep 17 16:44:40.840856 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:44:40.840830    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb3d00b_a6e9_4172_b79f_108213bc9d48.slice/crio-77d56a15f1fd935de08f2cc0c25f951cbabc1b6280eea175e385691993d226b6 WatchSource:0}: Error finding container 77d56a15f1fd935de08f2cc0c25f951cbabc1b6280eea175e385691993d226b6: Status 404 returned error can't find the container with id 77d56a15f1fd935de08f2cc0c25f951cbabc1b6280eea175e385691993d226b6
Sep 17 16:44:41.675782 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:41.675742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744b6569b4-7bpw8" event={"ID":"3eb3d00b-a6e9-4172-b79f-108213bc9d48","Type":"ContainerStarted","Data":"4c49b35e699a60e6f222b8bcecc8d9091751b5b68641babf5f7ef65ba42ad2e0"}
Sep 17 16:44:41.676154 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:41.675802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744b6569b4-7bpw8" event={"ID":"3eb3d00b-a6e9-4172-b79f-108213bc9d48","Type":"ContainerStarted","Data":"77d56a15f1fd935de08f2cc0c25f951cbabc1b6280eea175e385691993d226b6"}
Sep 17 16:44:41.695110 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:41.695064    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-744b6569b4-7bpw8" podStartSLOduration=1.695045222 podStartE2EDuration="1.695045222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:44:41.694301946 +0000 UTC m=+161.117239808" watchObservedRunningTime="2026-09-17 16:44:41.695045222 +0000 UTC m=+161.117983075"
Sep 17 16:44:50.562594 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:50.562494    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:50.562594 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:50.562565    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:50.567410 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:50.567386    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:44:50.723625 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:44:50.723601    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:47:02.697757 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:47:02.697724    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 16:47:02.700293 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:47:02.700266    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 16:47:02.706810 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:47:02.706789    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 16:48:04.663277 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.663188    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6745bf549f-srngd"]
Sep 17 16:48:04.665585 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.665564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.704874 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.704846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-trusted-ca-bundle\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.705035 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.704885    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-console-oauth-config\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.705035 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.704916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-console-config\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.705035 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.704993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-oauth-serving-cert\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.705193 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.705034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-service-ca\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.705193 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.705059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55qtl\" (UniqueName: \"kubernetes.io/projected/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-kube-api-access-55qtl\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.705193 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.705121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-console-serving-cert\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.718775 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.718746    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6745bf549f-srngd"]
Sep 17 16:48:04.806072 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.806046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-trusted-ca-bundle\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.806178 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.806078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-console-oauth-config\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.806178 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.806102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-console-config\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.806178 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.806122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-oauth-serving-cert\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.806178 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.806147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-service-ca\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.806178 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.806173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55qtl\" (UniqueName: \"kubernetes.io/projected/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-kube-api-access-55qtl\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.806403 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.806227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-console-serving-cert\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.806919 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.806895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-console-config\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.807057 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.807028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-oauth-serving-cert\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.807057 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.807000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-service-ca\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.807151 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.807070    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-trusted-ca-bundle\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.809324 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.809300    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-console-oauth-config\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.809395 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.809381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-console-serving-cert\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.817346 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.817317    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55qtl\" (UniqueName: \"kubernetes.io/projected/6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d-kube-api-access-55qtl\") pod \"console-6745bf549f-srngd\" (UID: \"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d\") " pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:04.975462 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:04.975429    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:05.248713 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:05.248637    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6745bf549f-srngd"]
Sep 17 16:48:05.251887 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:48:05.251859    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e76ee1e_ca4d_4af7_bf2d_8ea49d245f7d.slice/crio-92129679c76c6af0b3399d08fb3893bfa12ab78e899b1a4576288ff10b7f1fe7 WatchSource:0}: Error finding container 92129679c76c6af0b3399d08fb3893bfa12ab78e899b1a4576288ff10b7f1fe7: Status 404 returned error can't find the container with id 92129679c76c6af0b3399d08fb3893bfa12ab78e899b1a4576288ff10b7f1fe7
Sep 17 16:48:05.487047 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:05.487014    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6745bf549f-srngd" event={"ID":"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d","Type":"ContainerStarted","Data":"fcac31726888734693a6fc34937f631de38ca497b627531cd72df45f1eeb1bbe"}
Sep 17 16:48:05.487241 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:05.487063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6745bf549f-srngd" event={"ID":"6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d","Type":"ContainerStarted","Data":"92129679c76c6af0b3399d08fb3893bfa12ab78e899b1a4576288ff10b7f1fe7"}
Sep 17 16:48:05.507841 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:05.507759    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6745bf549f-srngd" podStartSLOduration=1.507745761 podStartE2EDuration="1.507745761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:48:05.5072712 +0000 UTC m=+364.930209065" watchObservedRunningTime="2026-09-17 16:48:05.507745761 +0000 UTC m=+364.930683623"
Sep 17 16:48:14.975925 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:14.975872    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:14.976488 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:14.975953    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:14.980609 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:14.980581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:15.537237 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:15.537206    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6745bf549f-srngd"
Sep 17 16:48:15.610945 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:15.610914    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-744b6569b4-7bpw8"]
Sep 17 16:48:40.631088 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.631014    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-744b6569b4-7bpw8" podUID="3eb3d00b-a6e9-4172-b79f-108213bc9d48" containerName="console" containerID="cri-o://4c49b35e699a60e6f222b8bcecc8d9091751b5b68641babf5f7ef65ba42ad2e0" gracePeriod=15
Sep 17 16:48:40.721142 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.721106    2581 patch_prober.go:28] interesting pod/console-744b6569b4-7bpw8 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.20:8443/health\": dial tcp 10.134.0.20:8443: connect: connection refused" start-of-body=
Sep 17 16:48:40.721252 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.721183    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-744b6569b4-7bpw8" podUID="3eb3d00b-a6e9-4172-b79f-108213bc9d48" containerName="console" probeResult="failure" output="Get \"https://10.134.0.20:8443/health\": dial tcp 10.134.0.20:8443: connect: connection refused"
Sep 17 16:48:40.875009 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.874982    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:48:40.971819 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.971793    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pjdz\" (UniqueName: \"kubernetes.io/projected/3eb3d00b-a6e9-4172-b79f-108213bc9d48-kube-api-access-5pjdz\") pod \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") "
Sep 17 16:48:40.971929 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.971826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-oauth-serving-cert\") pod \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") "
Sep 17 16:48:40.971929 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.971856    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-config\") pod \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") "
Sep 17 16:48:40.972032 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.972000    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-trusted-ca-bundle\") pod \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") "
Sep 17 16:48:40.972083 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.972042    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-service-ca\") pod \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") "
Sep 17 16:48:40.972142 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.972118    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-oauth-config\") pod \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") "
Sep 17 16:48:40.972198 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.972156    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-serving-cert\") pod \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\" (UID: \"3eb3d00b-a6e9-4172-b79f-108213bc9d48\") "
Sep 17 16:48:40.972544 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.972248    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3eb3d00b-a6e9-4172-b79f-108213bc9d48" (UID: "3eb3d00b-a6e9-4172-b79f-108213bc9d48"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:48:40.972544 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.972291    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-config" (OuterVolumeSpecName: "console-config") pod "3eb3d00b-a6e9-4172-b79f-108213bc9d48" (UID: "3eb3d00b-a6e9-4172-b79f-108213bc9d48"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:48:40.972544 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.972417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3eb3d00b-a6e9-4172-b79f-108213bc9d48" (UID: "3eb3d00b-a6e9-4172-b79f-108213bc9d48"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:48:40.972544 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.972472    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-service-ca" (OuterVolumeSpecName: "service-ca") pod "3eb3d00b-a6e9-4172-b79f-108213bc9d48" (UID: "3eb3d00b-a6e9-4172-b79f-108213bc9d48"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:48:40.972804 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.972753    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-oauth-serving-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:48:40.972804 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.972772    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-config\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:48:40.972804 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.972787    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-trusted-ca-bundle\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:48:40.972804 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.972800    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb3d00b-a6e9-4172-b79f-108213bc9d48-service-ca\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:48:40.974455 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.974431    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb3d00b-a6e9-4172-b79f-108213bc9d48-kube-api-access-5pjdz" (OuterVolumeSpecName: "kube-api-access-5pjdz") pod "3eb3d00b-a6e9-4172-b79f-108213bc9d48" (UID: "3eb3d00b-a6e9-4172-b79f-108213bc9d48"). InnerVolumeSpecName "kube-api-access-5pjdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:48:40.974455 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.974441    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3eb3d00b-a6e9-4172-b79f-108213bc9d48" (UID: "3eb3d00b-a6e9-4172-b79f-108213bc9d48"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:48:40.974691 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:40.974667    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3eb3d00b-a6e9-4172-b79f-108213bc9d48" (UID: "3eb3d00b-a6e9-4172-b79f-108213bc9d48"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:48:41.073569 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:41.073543    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-oauth-config\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:48:41.073569 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:41.073567    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3eb3d00b-a6e9-4172-b79f-108213bc9d48-console-serving-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:48:41.073691 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:41.073584    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pjdz\" (UniqueName: \"kubernetes.io/projected/3eb3d00b-a6e9-4172-b79f-108213bc9d48-kube-api-access-5pjdz\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:48:41.640790 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:41.640760    2581 generic.go:356] "Generic (PLEG): container finished" podID="3eb3d00b-a6e9-4172-b79f-108213bc9d48" containerID="4c49b35e699a60e6f222b8bcecc8d9091751b5b68641babf5f7ef65ba42ad2e0" exitCode=0
Sep 17 16:48:41.641200 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:41.640816    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744b6569b4-7bpw8"
Sep 17 16:48:41.641200 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:41.640847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744b6569b4-7bpw8" event={"ID":"3eb3d00b-a6e9-4172-b79f-108213bc9d48","Type":"ContainerDied","Data":"4c49b35e699a60e6f222b8bcecc8d9091751b5b68641babf5f7ef65ba42ad2e0"}
Sep 17 16:48:41.641200 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:41.640910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744b6569b4-7bpw8" event={"ID":"3eb3d00b-a6e9-4172-b79f-108213bc9d48","Type":"ContainerDied","Data":"77d56a15f1fd935de08f2cc0c25f951cbabc1b6280eea175e385691993d226b6"}
Sep 17 16:48:41.641200 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:41.640933    2581 scope.go:117] "RemoveContainer" containerID="4c49b35e699a60e6f222b8bcecc8d9091751b5b68641babf5f7ef65ba42ad2e0"
Sep 17 16:48:41.650530 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:41.650506    2581 scope.go:117] "RemoveContainer" containerID="4c49b35e699a60e6f222b8bcecc8d9091751b5b68641babf5f7ef65ba42ad2e0"
Sep 17 16:48:41.650856 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:48:41.650834    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c49b35e699a60e6f222b8bcecc8d9091751b5b68641babf5f7ef65ba42ad2e0\": container with ID starting with 4c49b35e699a60e6f222b8bcecc8d9091751b5b68641babf5f7ef65ba42ad2e0 not found: ID does not exist" containerID="4c49b35e699a60e6f222b8bcecc8d9091751b5b68641babf5f7ef65ba42ad2e0"
Sep 17 16:48:41.650917 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:41.650865    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c49b35e699a60e6f222b8bcecc8d9091751b5b68641babf5f7ef65ba42ad2e0"} err="failed to get container status \"4c49b35e699a60e6f222b8bcecc8d9091751b5b68641babf5f7ef65ba42ad2e0\": rpc error: code = NotFound desc = could not find container \"4c49b35e699a60e6f222b8bcecc8d9091751b5b68641babf5f7ef65ba42ad2e0\": container with ID starting with 4c49b35e699a60e6f222b8bcecc8d9091751b5b68641babf5f7ef65ba42ad2e0 not found: ID does not exist"
Sep 17 16:48:41.667653 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:41.667621    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-744b6569b4-7bpw8"]
Sep 17 16:48:41.673458 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:41.673435    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-744b6569b4-7bpw8"]
Sep 17 16:48:42.839670 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:48:42.835899    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eb3d00b-a6e9-4172-b79f-108213bc9d48" path="/var/lib/kubelet/pods/3eb3d00b-a6e9-4172-b79f-108213bc9d48/volumes"
Sep 17 16:50:13.238586 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.238553    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"]
Sep 17 16:50:13.239055 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.239006    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb3d00b-a6e9-4172-b79f-108213bc9d48" containerName="console"
Sep 17 16:50:13.239055 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.239021    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb3d00b-a6e9-4172-b79f-108213bc9d48" containerName="console"
Sep 17 16:50:13.239137 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.239113    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb3d00b-a6e9-4172-b79f-108213bc9d48" containerName="console"
Sep 17 16:50:13.241245 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.241228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"
Sep 17 16:50:13.247105 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.247077    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 16:50:13.247238 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.247158    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 16:50:13.248015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.247993    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 16:50:13.248015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.248009    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-nh2lq\""
Sep 17 16:50:13.248173 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.248083    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 16:50:13.251046 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.251030    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 16:50:13.261000 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.260980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"]
Sep 17 16:50:13.334020 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.333997    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kbjz\" (UniqueName: \"kubernetes.io/projected/9ff80221-539c-4c1b-9ecf-4b3df4828f14-kube-api-access-4kbjz\") pod \"keda-metrics-apiserver-77857b7b98-x264f\" (UID: \"9ff80221-539c-4c1b-9ecf-4b3df4828f14\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"
Sep 17 16:50:13.334122 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.334033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9ff80221-539c-4c1b-9ecf-4b3df4828f14-certificates\") pod \"keda-metrics-apiserver-77857b7b98-x264f\" (UID: \"9ff80221-539c-4c1b-9ecf-4b3df4828f14\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"
Sep 17 16:50:13.334167 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.334139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9ff80221-539c-4c1b-9ecf-4b3df4828f14-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-x264f\" (UID: \"9ff80221-539c-4c1b-9ecf-4b3df4828f14\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"
Sep 17 16:50:13.435352 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.435332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9ff80221-539c-4c1b-9ecf-4b3df4828f14-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-x264f\" (UID: \"9ff80221-539c-4c1b-9ecf-4b3df4828f14\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"
Sep 17 16:50:13.435461 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.435378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kbjz\" (UniqueName: \"kubernetes.io/projected/9ff80221-539c-4c1b-9ecf-4b3df4828f14-kube-api-access-4kbjz\") pod \"keda-metrics-apiserver-77857b7b98-x264f\" (UID: \"9ff80221-539c-4c1b-9ecf-4b3df4828f14\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"
Sep 17 16:50:13.435461 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.435395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9ff80221-539c-4c1b-9ecf-4b3df4828f14-certificates\") pod \"keda-metrics-apiserver-77857b7b98-x264f\" (UID: \"9ff80221-539c-4c1b-9ecf-4b3df4828f14\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"
Sep 17 16:50:13.435551 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:50:13.435489    2581 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 17 16:50:13.435551 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:50:13.435503    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-x264f: secret "keda-metrics-apiserver-certs" not found
Sep 17 16:50:13.435551 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:50:13.435547    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ff80221-539c-4c1b-9ecf-4b3df4828f14-certificates podName:9ff80221-539c-4c1b-9ecf-4b3df4828f14 nodeName:}" failed. No retries permitted until 2026-09-17 16:50:13.935533591 +0000 UTC m=+493.358471432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9ff80221-539c-4c1b-9ecf-4b3df4828f14-certificates") pod "keda-metrics-apiserver-77857b7b98-x264f" (UID: "9ff80221-539c-4c1b-9ecf-4b3df4828f14") : secret "keda-metrics-apiserver-certs" not found
Sep 17 16:50:13.435749 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.435729    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9ff80221-539c-4c1b-9ecf-4b3df4828f14-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-x264f\" (UID: \"9ff80221-539c-4c1b-9ecf-4b3df4828f14\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"
Sep 17 16:50:13.482151 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.482127    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kbjz\" (UniqueName: \"kubernetes.io/projected/9ff80221-539c-4c1b-9ecf-4b3df4828f14-kube-api-access-4kbjz\") pod \"keda-metrics-apiserver-77857b7b98-x264f\" (UID: \"9ff80221-539c-4c1b-9ecf-4b3df4828f14\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"
Sep 17 16:50:13.939193 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.939064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9ff80221-539c-4c1b-9ecf-4b3df4828f14-certificates\") pod \"keda-metrics-apiserver-77857b7b98-x264f\" (UID: \"9ff80221-539c-4c1b-9ecf-4b3df4828f14\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"
Sep 17 16:50:13.941501 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:13.941476    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9ff80221-539c-4c1b-9ecf-4b3df4828f14-certificates\") pod \"keda-metrics-apiserver-77857b7b98-x264f\" (UID: \"9ff80221-539c-4c1b-9ecf-4b3df4828f14\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"
Sep 17 16:50:14.151889 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:14.151859    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"
Sep 17 16:50:14.432956 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:14.432927    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"]
Sep 17 16:50:14.436394 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:50:14.436369    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff80221_539c_4c1b_9ecf_4b3df4828f14.slice/crio-83a782a198f7955ab48500816546ef89b6bef1b7f269b4f7edce7266e7c9c92a WatchSource:0}: Error finding container 83a782a198f7955ab48500816546ef89b6bef1b7f269b4f7edce7266e7c9c92a: Status 404 returned error can't find the container with id 83a782a198f7955ab48500816546ef89b6bef1b7f269b4f7edce7266e7c9c92a
Sep 17 16:50:14.437646 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:14.437631    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:50:15.007750 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:15.007719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f" event={"ID":"9ff80221-539c-4c1b-9ecf-4b3df4828f14","Type":"ContainerStarted","Data":"83a782a198f7955ab48500816546ef89b6bef1b7f269b4f7edce7266e7c9c92a"}
Sep 17 16:50:18.022281 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:18.022241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f" event={"ID":"9ff80221-539c-4c1b-9ecf-4b3df4828f14","Type":"ContainerStarted","Data":"6ee5d28dd9ee1356706de34efd3086bd09958caede144c49eb6727187e5a3473"}
Sep 17 16:50:18.022669 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:18.022385    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"
Sep 17 16:50:18.044834 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:18.044783    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f" podStartSLOduration=2.183239646 podStartE2EDuration="5.044764821s" totalImagesPullingTime="2.861525175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:50:18.043194037 +0000 UTC m=+497.466131901" watchObservedRunningTime="2026-09-17 16:50:18.044764821 +0000 UTC m=+497.467702687"
Sep 17 16:50:29.031342 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:50:29.031314    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x264f"
Sep 17 16:51:20.410403 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:20.410331    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xjwm5"]
Sep 17 16:51:20.413298 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:20.413278    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xjwm5"
Sep 17 16:51:20.415482 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:20.415459    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 16:51:20.415610 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:20.415466    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 16:51:20.416315 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:20.416301    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jxhw8\""
Sep 17 16:51:20.424842 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:20.424816    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xjwm5"]
Sep 17 16:51:20.516599 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:20.516573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g8wv\" (UniqueName: \"kubernetes.io/projected/a8e73281-ef23-4c70-8ca4-f5c32be4145a-kube-api-access-2g8wv\") pod \"cert-manager-webhook-6486ffd997-xjwm5\" (UID: \"a8e73281-ef23-4c70-8ca4-f5c32be4145a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xjwm5"
Sep 17 16:51:20.516713 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:20.516604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8e73281-ef23-4c70-8ca4-f5c32be4145a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xjwm5\" (UID: \"a8e73281-ef23-4c70-8ca4-f5c32be4145a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xjwm5"
Sep 17 16:51:20.616937 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:20.616915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g8wv\" (UniqueName: \"kubernetes.io/projected/a8e73281-ef23-4c70-8ca4-f5c32be4145a-kube-api-access-2g8wv\") pod \"cert-manager-webhook-6486ffd997-xjwm5\" (UID: \"a8e73281-ef23-4c70-8ca4-f5c32be4145a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xjwm5"
Sep 17 16:51:20.617042 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:20.616944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8e73281-ef23-4c70-8ca4-f5c32be4145a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xjwm5\" (UID: \"a8e73281-ef23-4c70-8ca4-f5c32be4145a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xjwm5"
Sep 17 16:51:20.626858 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:20.626840    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g8wv\" (UniqueName: \"kubernetes.io/projected/a8e73281-ef23-4c70-8ca4-f5c32be4145a-kube-api-access-2g8wv\") pod \"cert-manager-webhook-6486ffd997-xjwm5\" (UID: \"a8e73281-ef23-4c70-8ca4-f5c32be4145a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xjwm5"
Sep 17 16:51:20.627060 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:20.627044    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8e73281-ef23-4c70-8ca4-f5c32be4145a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xjwm5\" (UID: \"a8e73281-ef23-4c70-8ca4-f5c32be4145a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xjwm5"
Sep 17 16:51:20.733048 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:20.733025    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xjwm5"
Sep 17 16:51:21.006867 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:21.006838    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xjwm5"]
Sep 17 16:51:21.008498 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:51:21.008472    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8e73281_ef23_4c70_8ca4_f5c32be4145a.slice/crio-79825b902b29ff5f71adf0acf0d6b49aee85e67d10fbd310d8b10ddbb7237cbd WatchSource:0}: Error finding container 79825b902b29ff5f71adf0acf0d6b49aee85e67d10fbd310d8b10ddbb7237cbd: Status 404 returned error can't find the container with id 79825b902b29ff5f71adf0acf0d6b49aee85e67d10fbd310d8b10ddbb7237cbd
Sep 17 16:51:21.274189 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:21.274125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xjwm5" event={"ID":"a8e73281-ef23-4c70-8ca4-f5c32be4145a","Type":"ContainerStarted","Data":"79825b902b29ff5f71adf0acf0d6b49aee85e67d10fbd310d8b10ddbb7237cbd"}
Sep 17 16:51:22.969390 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:22.969358    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ctjmw"]
Sep 17 16:51:22.973125 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:22.973074    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ctjmw"
Sep 17 16:51:22.985321 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:22.985297    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ctjmw"]
Sep 17 16:51:23.034555 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:23.034529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c4c474c-e837-49b4-9471-cb18d0669eeb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ctjmw\" (UID: \"0c4c474c-e837-49b4-9471-cb18d0669eeb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ctjmw"
Sep 17 16:51:23.034674 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:23.034563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf2gl\" (UniqueName: \"kubernetes.io/projected/0c4c474c-e837-49b4-9471-cb18d0669eeb-kube-api-access-tf2gl\") pod \"cert-manager-cainjector-559cdd8c87-ctjmw\" (UID: \"0c4c474c-e837-49b4-9471-cb18d0669eeb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ctjmw"
Sep 17 16:51:23.135328 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:23.135304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf2gl\" (UniqueName: \"kubernetes.io/projected/0c4c474c-e837-49b4-9471-cb18d0669eeb-kube-api-access-tf2gl\") pod \"cert-manager-cainjector-559cdd8c87-ctjmw\" (UID: \"0c4c474c-e837-49b4-9471-cb18d0669eeb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ctjmw"
Sep 17 16:51:23.135458 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:23.135380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c4c474c-e837-49b4-9471-cb18d0669eeb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ctjmw\" (UID: \"0c4c474c-e837-49b4-9471-cb18d0669eeb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ctjmw"
Sep 17 16:51:23.151269 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:23.151244    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf2gl\" (UniqueName: \"kubernetes.io/projected/0c4c474c-e837-49b4-9471-cb18d0669eeb-kube-api-access-tf2gl\") pod \"cert-manager-cainjector-559cdd8c87-ctjmw\" (UID: \"0c4c474c-e837-49b4-9471-cb18d0669eeb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ctjmw"
Sep 17 16:51:23.152532 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:23.152511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c4c474c-e837-49b4-9471-cb18d0669eeb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ctjmw\" (UID: \"0c4c474c-e837-49b4-9471-cb18d0669eeb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ctjmw"
Sep 17 16:51:23.285710 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:23.285641    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ctjmw"
Sep 17 16:51:24.284901 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:24.284876    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ctjmw"]
Sep 17 16:51:24.287052 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:51:24.287019    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c4c474c_e837_49b4_9471_cb18d0669eeb.slice/crio-9412dd9388a45cdae527f4c4f88946d098b233e40dbcb6be22cf7b776e20691f WatchSource:0}: Error finding container 9412dd9388a45cdae527f4c4f88946d098b233e40dbcb6be22cf7b776e20691f: Status 404 returned error can't find the container with id 9412dd9388a45cdae527f4c4f88946d098b233e40dbcb6be22cf7b776e20691f
Sep 17 16:51:24.289722 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:24.289692    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xjwm5" event={"ID":"a8e73281-ef23-4c70-8ca4-f5c32be4145a","Type":"ContainerStarted","Data":"3a7a2d3ebf7fe346294ad4594fa3a24a987d85dd96dd7def9d43f22d1dad012b"}
Sep 17 16:51:24.289819 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:24.289751    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xjwm5"
Sep 17 16:51:24.318076 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:24.318020    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-xjwm5" podStartSLOduration=1.25217018 podStartE2EDuration="4.318000215s" totalImagesPullingTime="3.065830035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:51:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:51:24.317944854 +0000 UTC m=+563.740882720" watchObservedRunningTime="2026-09-17 16:51:24.318000215 +0000 UTC m=+563.740938101"
Sep 17 16:51:25.296610 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:25.296573    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ctjmw" event={"ID":"0c4c474c-e837-49b4-9471-cb18d0669eeb","Type":"ContainerStarted","Data":"407e40241d7447bdd565c93daddeb2e506b6805e39fec85c367c6b67b9eb7bd4"}
Sep 17 16:51:25.296610 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:25.296616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ctjmw" event={"ID":"0c4c474c-e837-49b4-9471-cb18d0669eeb","Type":"ContainerStarted","Data":"9412dd9388a45cdae527f4c4f88946d098b233e40dbcb6be22cf7b776e20691f"}
Sep 17 16:51:25.323041 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:25.322992    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ctjmw" podStartSLOduration=3.322977605 podStartE2EDuration="3.322977605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:51:25.32106318 +0000 UTC m=+564.744001044" watchObservedRunningTime="2026-09-17 16:51:25.322977605 +0000 UTC m=+564.745915460"
Sep 17 16:51:30.298702 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:30.298664    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xjwm5"
Sep 17 16:51:48.097128 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.097092    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"]
Sep 17 16:51:48.100311 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.100294    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:48.103675 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.103641    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:51:48.103675 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.103642    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 16:51:48.103879 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.103705    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 16:51:48.103879 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.103724    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 16:51:48.103879 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.103824    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wp4cr\""
Sep 17 16:51:48.103879 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.103869    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 16:51:48.111529 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.111510    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"]
Sep 17 16:51:48.212883 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.212860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp672\" (UniqueName: \"kubernetes.io/projected/d08f318a-6fc4-42aa-a120-aad03695f9ac-kube-api-access-dp672\") pod \"lws-controller-manager-b8d4866b4-vflkm\" (UID: \"d08f318a-6fc4-42aa-a120-aad03695f9ac\") " pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:48.213015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.212905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d08f318a-6fc4-42aa-a120-aad03695f9ac-cert\") pod \"lws-controller-manager-b8d4866b4-vflkm\" (UID: \"d08f318a-6fc4-42aa-a120-aad03695f9ac\") " pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:48.213015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.212959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d08f318a-6fc4-42aa-a120-aad03695f9ac-manager-config\") pod \"lws-controller-manager-b8d4866b4-vflkm\" (UID: \"d08f318a-6fc4-42aa-a120-aad03695f9ac\") " pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:48.213015 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.213005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d08f318a-6fc4-42aa-a120-aad03695f9ac-metrics-cert\") pod \"lws-controller-manager-b8d4866b4-vflkm\" (UID: \"d08f318a-6fc4-42aa-a120-aad03695f9ac\") " pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:48.314072 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.314050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d08f318a-6fc4-42aa-a120-aad03695f9ac-metrics-cert\") pod \"lws-controller-manager-b8d4866b4-vflkm\" (UID: \"d08f318a-6fc4-42aa-a120-aad03695f9ac\") " pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:48.314171 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.314091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp672\" (UniqueName: \"kubernetes.io/projected/d08f318a-6fc4-42aa-a120-aad03695f9ac-kube-api-access-dp672\") pod \"lws-controller-manager-b8d4866b4-vflkm\" (UID: \"d08f318a-6fc4-42aa-a120-aad03695f9ac\") " pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:48.314171 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.314127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d08f318a-6fc4-42aa-a120-aad03695f9ac-cert\") pod \"lws-controller-manager-b8d4866b4-vflkm\" (UID: \"d08f318a-6fc4-42aa-a120-aad03695f9ac\") " pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:48.314171 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.314159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d08f318a-6fc4-42aa-a120-aad03695f9ac-manager-config\") pod \"lws-controller-manager-b8d4866b4-vflkm\" (UID: \"d08f318a-6fc4-42aa-a120-aad03695f9ac\") " pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:48.314738 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.314721    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d08f318a-6fc4-42aa-a120-aad03695f9ac-manager-config\") pod \"lws-controller-manager-b8d4866b4-vflkm\" (UID: \"d08f318a-6fc4-42aa-a120-aad03695f9ac\") " pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:48.316668 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.316648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d08f318a-6fc4-42aa-a120-aad03695f9ac-cert\") pod \"lws-controller-manager-b8d4866b4-vflkm\" (UID: \"d08f318a-6fc4-42aa-a120-aad03695f9ac\") " pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:48.316668 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.316654    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d08f318a-6fc4-42aa-a120-aad03695f9ac-metrics-cert\") pod \"lws-controller-manager-b8d4866b4-vflkm\" (UID: \"d08f318a-6fc4-42aa-a120-aad03695f9ac\") " pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:48.326186 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.326165    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp672\" (UniqueName: \"kubernetes.io/projected/d08f318a-6fc4-42aa-a120-aad03695f9ac-kube-api-access-dp672\") pod \"lws-controller-manager-b8d4866b4-vflkm\" (UID: \"d08f318a-6fc4-42aa-a120-aad03695f9ac\") " pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:48.409790 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.409737    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:48.683651 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:48.683564    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"]
Sep 17 16:51:48.686394 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:51:48.686363    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08f318a_6fc4_42aa_a120_aad03695f9ac.slice/crio-d8674ff6ab92798774ff38881805671832b764349ce1f3242769e1fe69707106 WatchSource:0}: Error finding container d8674ff6ab92798774ff38881805671832b764349ce1f3242769e1fe69707106: Status 404 returned error can't find the container with id d8674ff6ab92798774ff38881805671832b764349ce1f3242769e1fe69707106
Sep 17 16:51:49.399144 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:49.399074    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm" event={"ID":"d08f318a-6fc4-42aa-a120-aad03695f9ac","Type":"ContainerStarted","Data":"d8674ff6ab92798774ff38881805671832b764349ce1f3242769e1fe69707106"}
Sep 17 16:51:51.410498 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:51.410412    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm" event={"ID":"d08f318a-6fc4-42aa-a120-aad03695f9ac","Type":"ContainerStarted","Data":"d32d7ac150ea013d1f45af89ca96a6d9c1174f8d980d9017a158a4861ff022f4"}
Sep 17 16:51:51.410868 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:51.410543    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:51:51.432340 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:51:51.432292    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm" podStartSLOduration=1.160952345 podStartE2EDuration="3.432278802s" totalImagesPullingTime="2.271326457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:51:51.431550966 +0000 UTC m=+590.854488830" watchObservedRunningTime="2026-09-17 16:51:51.432278802 +0000 UTC m=+590.855216665"
Sep 17 16:52:02.416777 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:02.416743    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-b8d4866b4-vflkm"
Sep 17 16:52:02.734245 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:02.734218    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 16:52:02.735267 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:02.735244    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 16:52:36.600064 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:36.600006    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6l266"]
Sep 17 16:52:36.609503 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:36.609468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6l266"
Sep 17 16:52:36.612131 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:36.612106    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 16:52:36.612241 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:36.612106    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9xwcf\""
Sep 17 16:52:36.612888 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:36.612823    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 16:52:36.612888 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:36.612829    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 17 16:52:36.615918 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:36.615895    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6l266"]
Sep 17 16:52:36.797184 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:36.797133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5tdz\" (UniqueName: \"kubernetes.io/projected/bd12b862-34c9-4adc-b3d3-178f564aea1e-kube-api-access-l5tdz\") pod \"dns-operator-controller-manager-65dc8f76f5-6l266\" (UID: \"bd12b862-34c9-4adc-b3d3-178f564aea1e\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6l266"
Sep 17 16:52:36.898431 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:36.898329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5tdz\" (UniqueName: \"kubernetes.io/projected/bd12b862-34c9-4adc-b3d3-178f564aea1e-kube-api-access-l5tdz\") pod \"dns-operator-controller-manager-65dc8f76f5-6l266\" (UID: \"bd12b862-34c9-4adc-b3d3-178f564aea1e\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6l266"
Sep 17 16:52:36.914306 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:36.914278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5tdz\" (UniqueName: \"kubernetes.io/projected/bd12b862-34c9-4adc-b3d3-178f564aea1e-kube-api-access-l5tdz\") pod \"dns-operator-controller-manager-65dc8f76f5-6l266\" (UID: \"bd12b862-34c9-4adc-b3d3-178f564aea1e\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6l266"
Sep 17 16:52:36.921018 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:36.920992    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6l266"
Sep 17 16:52:37.212460 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:37.212435    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6l266"]
Sep 17 16:52:37.215200 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:52:37.215173    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd12b862_34c9_4adc_b3d3_178f564aea1e.slice/crio-6197cbc9af7dad0d6b3c0b0fe0350e297c29470a27eea5419bc7d0e8b9b96617 WatchSource:0}: Error finding container 6197cbc9af7dad0d6b3c0b0fe0350e297c29470a27eea5419bc7d0e8b9b96617: Status 404 returned error can't find the container with id 6197cbc9af7dad0d6b3c0b0fe0350e297c29470a27eea5419bc7d0e8b9b96617
Sep 17 16:52:37.596961 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:37.596883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6l266" event={"ID":"bd12b862-34c9-4adc-b3d3-178f564aea1e","Type":"ContainerStarted","Data":"6197cbc9af7dad0d6b3c0b0fe0350e297c29470a27eea5419bc7d0e8b9b96617"}
Sep 17 16:52:39.607638 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:39.607604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6l266" event={"ID":"bd12b862-34c9-4adc-b3d3-178f564aea1e","Type":"ContainerStarted","Data":"3338fc5e26010a192902adf9020bcea0d203326ea744f75e7f86ae708f692cd3"}
Sep 17 16:52:39.608029 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:39.607737    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6l266"
Sep 17 16:52:39.651628 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:39.651584    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6l266" podStartSLOduration=1.94617734 podStartE2EDuration="3.651569979s" totalImagesPullingTime="1.705392639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:52:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:52:39.649112038 +0000 UTC m=+639.072049900" watchObservedRunningTime="2026-09-17 16:52:39.651569979 +0000 UTC m=+639.074507841"
Sep 17 16:52:43.650873 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:43.650830    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc"]
Sep 17 16:52:43.654273 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:43.654256    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc"
Sep 17 16:52:43.687857 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:43.687825    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc"]
Sep 17 16:52:43.738451 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:43.738413    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9n55z\""
Sep 17 16:52:43.751694 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:43.751656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlqt5\" (UniqueName: \"kubernetes.io/projected/8b987118-e74d-4e45-b9f2-4587471513a2-kube-api-access-xlqt5\") pod \"kuadrant-operator-controller-manager-6d4cb79c-prrwc\" (UID: \"8b987118-e74d-4e45-b9f2-4587471513a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc"
Sep 17 16:52:43.751836 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:43.751775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b987118-e74d-4e45-b9f2-4587471513a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-prrwc\" (UID: \"8b987118-e74d-4e45-b9f2-4587471513a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc"
Sep 17 16:52:43.853042 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:43.853004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlqt5\" (UniqueName: \"kubernetes.io/projected/8b987118-e74d-4e45-b9f2-4587471513a2-kube-api-access-xlqt5\") pod \"kuadrant-operator-controller-manager-6d4cb79c-prrwc\" (UID: \"8b987118-e74d-4e45-b9f2-4587471513a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc"
Sep 17 16:52:43.853239 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:43.853069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b987118-e74d-4e45-b9f2-4587471513a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-prrwc\" (UID: \"8b987118-e74d-4e45-b9f2-4587471513a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc"
Sep 17 16:52:43.853473 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:43.853450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b987118-e74d-4e45-b9f2-4587471513a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-prrwc\" (UID: \"8b987118-e74d-4e45-b9f2-4587471513a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc"
Sep 17 16:52:43.866497 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:43.866470    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlqt5\" (UniqueName: \"kubernetes.io/projected/8b987118-e74d-4e45-b9f2-4587471513a2-kube-api-access-xlqt5\") pod \"kuadrant-operator-controller-manager-6d4cb79c-prrwc\" (UID: \"8b987118-e74d-4e45-b9f2-4587471513a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc"
Sep 17 16:52:43.964402 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:43.964379    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc"
Sep 17 16:52:44.246136 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:44.246054    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc"]
Sep 17 16:52:44.248427 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:52:44.248382    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b987118_e74d_4e45_b9f2_4587471513a2.slice/crio-538d1a9e3328e5ba4ca1b48dbacf72feebf2c3949868eb1e58c0fd5a0601014c WatchSource:0}: Error finding container 538d1a9e3328e5ba4ca1b48dbacf72feebf2c3949868eb1e58c0fd5a0601014c: Status 404 returned error can't find the container with id 538d1a9e3328e5ba4ca1b48dbacf72feebf2c3949868eb1e58c0fd5a0601014c
Sep 17 16:52:44.629857 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:44.629773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc" event={"ID":"8b987118-e74d-4e45-b9f2-4587471513a2","Type":"ContainerStarted","Data":"538d1a9e3328e5ba4ca1b48dbacf72feebf2c3949868eb1e58c0fd5a0601014c"}
Sep 17 16:52:48.650669 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:48.650623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc" event={"ID":"8b987118-e74d-4e45-b9f2-4587471513a2","Type":"ContainerStarted","Data":"e5cb2f0e138832f1872f7377692ea5309cfdf879178188df9befb7aa17c14162"}
Sep 17 16:52:48.651079 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:48.650733    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc"
Sep 17 16:52:48.673396 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:48.673337    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc" podStartSLOduration=2.08846986 podStartE2EDuration="5.673312131s" totalImagesPullingTime="3.584842271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:52:48.672668513 +0000 UTC m=+648.095606376" watchObservedRunningTime="2026-09-17 16:52:48.673312131 +0000 UTC m=+648.096249996"
Sep 17 16:52:50.614568 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:50.614532    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-6l266"
Sep 17 16:52:59.658377 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:52:59.658342    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-prrwc"
Sep 17 16:53:32.376955 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.376907    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8p9sc"]
Sep 17 16:53:32.379593 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.379568    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8p9sc"
Sep 17 16:53:32.382030 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.382005    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5df55\""
Sep 17 16:53:32.389868 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.389846    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8p9sc"]
Sep 17 16:53:32.511272 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.511247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx7sh\" (UniqueName: \"kubernetes.io/projected/415a2b92-8c2a-4aa3-b925-cc125557c202-kube-api-access-bx7sh\") pod \"authorino-7bcfd8f9cd-8p9sc\" (UID: \"415a2b92-8c2a-4aa3-b925-cc125557c202\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8p9sc"
Sep 17 16:53:32.612193 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.612168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx7sh\" (UniqueName: \"kubernetes.io/projected/415a2b92-8c2a-4aa3-b925-cc125557c202-kube-api-access-bx7sh\") pod \"authorino-7bcfd8f9cd-8p9sc\" (UID: \"415a2b92-8c2a-4aa3-b925-cc125557c202\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8p9sc"
Sep 17 16:53:32.628691 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.628638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx7sh\" (UniqueName: \"kubernetes.io/projected/415a2b92-8c2a-4aa3-b925-cc125557c202-kube-api-access-bx7sh\") pod \"authorino-7bcfd8f9cd-8p9sc\" (UID: \"415a2b92-8c2a-4aa3-b925-cc125557c202\") " pod="kuadrant-system/authorino-7bcfd8f9cd-8p9sc"
Sep 17 16:53:32.665888 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.665864    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-dfs95"]
Sep 17 16:53:32.668242 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.668224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dfs95"
Sep 17 16:53:32.684779 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.684758    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dfs95"]
Sep 17 16:53:32.689012 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.688994    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8p9sc"
Sep 17 16:53:32.813809 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.813782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdkjk\" (UniqueName: \"kubernetes.io/projected/2dae6917-b2a3-46c1-9505-03fd253c93a5-kube-api-access-fdkjk\") pod \"authorino-59c974bd48-dfs95\" (UID: \"2dae6917-b2a3-46c1-9505-03fd253c93a5\") " pod="kuadrant-system/authorino-59c974bd48-dfs95"
Sep 17 16:53:32.915111 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.915055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdkjk\" (UniqueName: \"kubernetes.io/projected/2dae6917-b2a3-46c1-9505-03fd253c93a5-kube-api-access-fdkjk\") pod \"authorino-59c974bd48-dfs95\" (UID: \"2dae6917-b2a3-46c1-9505-03fd253c93a5\") " pod="kuadrant-system/authorino-59c974bd48-dfs95"
Sep 17 16:53:32.927560 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.927534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdkjk\" (UniqueName: \"kubernetes.io/projected/2dae6917-b2a3-46c1-9505-03fd253c93a5-kube-api-access-fdkjk\") pod \"authorino-59c974bd48-dfs95\" (UID: \"2dae6917-b2a3-46c1-9505-03fd253c93a5\") " pod="kuadrant-system/authorino-59c974bd48-dfs95"
Sep 17 16:53:32.960945 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.960926    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8p9sc"]
Sep 17 16:53:32.963191 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:53:32.963162    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod415a2b92_8c2a_4aa3_b925_cc125557c202.slice/crio-2dac1953e7a0eb553b6b1ea3b432e03beeea8d38d6b300adcd48e49e4cfdef49 WatchSource:0}: Error finding container 2dac1953e7a0eb553b6b1ea3b432e03beeea8d38d6b300adcd48e49e4cfdef49: Status 404 returned error can't find the container with id 2dac1953e7a0eb553b6b1ea3b432e03beeea8d38d6b300adcd48e49e4cfdef49
Sep 17 16:53:32.977414 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:32.977397    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dfs95"
Sep 17 16:53:33.246635 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:33.246602    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dfs95"]
Sep 17 16:53:33.248318 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:53:33.248293    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dae6917_b2a3_46c1_9505_03fd253c93a5.slice/crio-0c47c6c5af2ebc9231a4b1844ef838fe604c611caf2b9f6a5cb82d8e32da8a82 WatchSource:0}: Error finding container 0c47c6c5af2ebc9231a4b1844ef838fe604c611caf2b9f6a5cb82d8e32da8a82: Status 404 returned error can't find the container with id 0c47c6c5af2ebc9231a4b1844ef838fe604c611caf2b9f6a5cb82d8e32da8a82
Sep 17 16:53:33.871594 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:33.871554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dfs95" event={"ID":"2dae6917-b2a3-46c1-9505-03fd253c93a5","Type":"ContainerStarted","Data":"0c47c6c5af2ebc9231a4b1844ef838fe604c611caf2b9f6a5cb82d8e32da8a82"}
Sep 17 16:53:33.872807 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:33.872774    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8p9sc" event={"ID":"415a2b92-8c2a-4aa3-b925-cc125557c202","Type":"ContainerStarted","Data":"2dac1953e7a0eb553b6b1ea3b432e03beeea8d38d6b300adcd48e49e4cfdef49"}
Sep 17 16:53:35.888042 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:35.888002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dfs95" event={"ID":"2dae6917-b2a3-46c1-9505-03fd253c93a5","Type":"ContainerStarted","Data":"06f836a540bba621888e96614abeeea3f2c2a0c8fe028c0f531b582f163bb2b9"}
Sep 17 16:53:35.889694 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:35.889665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8p9sc" event={"ID":"415a2b92-8c2a-4aa3-b925-cc125557c202","Type":"ContainerStarted","Data":"f929fd62f596510f3dad42eb800f891f55f8317f03c4bd0b21b6243ed1f4a000"}
Sep 17 16:53:35.912483 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:35.912437    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-dfs95" podStartSLOduration=1.92398639 podStartE2EDuration="3.912424894s" totalImagesPullingTime="1.988438504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:53:35.910010809 +0000 UTC m=+695.332948675" watchObservedRunningTime="2026-09-17 16:53:35.912424894 +0000 UTC m=+695.335362757"
Sep 17 16:53:35.928553 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:35.928511    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-8p9sc" podStartSLOduration=1.6496454310000002 podStartE2EDuration="3.928497029s" totalImagesPullingTime="2.278851598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:53:35.927701268 +0000 UTC m=+695.350639133" watchObservedRunningTime="2026-09-17 16:53:35.928497029 +0000 UTC m=+695.351434891"
Sep 17 16:53:35.967992 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:35.967954    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8p9sc"]
Sep 17 16:53:37.898094 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:37.898042    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-8p9sc" podUID="415a2b92-8c2a-4aa3-b925-cc125557c202" containerName="authorino" containerID="cri-o://f929fd62f596510f3dad42eb800f891f55f8317f03c4bd0b21b6243ed1f4a000" gracePeriod=30
Sep 17 16:53:38.150661 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:38.150600    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8p9sc"
Sep 17 16:53:38.255604 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:38.255577    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx7sh\" (UniqueName: \"kubernetes.io/projected/415a2b92-8c2a-4aa3-b925-cc125557c202-kube-api-access-bx7sh\") pod \"415a2b92-8c2a-4aa3-b925-cc125557c202\" (UID: \"415a2b92-8c2a-4aa3-b925-cc125557c202\") "
Sep 17 16:53:38.257626 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:38.257604    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415a2b92-8c2a-4aa3-b925-cc125557c202-kube-api-access-bx7sh" (OuterVolumeSpecName: "kube-api-access-bx7sh") pod "415a2b92-8c2a-4aa3-b925-cc125557c202" (UID: "415a2b92-8c2a-4aa3-b925-cc125557c202"). InnerVolumeSpecName "kube-api-access-bx7sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:53:38.356077 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:38.356052    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx7sh\" (UniqueName: \"kubernetes.io/projected/415a2b92-8c2a-4aa3-b925-cc125557c202-kube-api-access-bx7sh\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:53:38.904049 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:38.904011    2581 generic.go:356] "Generic (PLEG): container finished" podID="415a2b92-8c2a-4aa3-b925-cc125557c202" containerID="f929fd62f596510f3dad42eb800f891f55f8317f03c4bd0b21b6243ed1f4a000" exitCode=0
Sep 17 16:53:38.904438 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:38.904054    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-8p9sc"
Sep 17 16:53:38.904438 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:38.904069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8p9sc" event={"ID":"415a2b92-8c2a-4aa3-b925-cc125557c202","Type":"ContainerDied","Data":"f929fd62f596510f3dad42eb800f891f55f8317f03c4bd0b21b6243ed1f4a000"}
Sep 17 16:53:38.904438 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:38.904104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-8p9sc" event={"ID":"415a2b92-8c2a-4aa3-b925-cc125557c202","Type":"ContainerDied","Data":"2dac1953e7a0eb553b6b1ea3b432e03beeea8d38d6b300adcd48e49e4cfdef49"}
Sep 17 16:53:38.904438 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:38.904125    2581 scope.go:117] "RemoveContainer" containerID="f929fd62f596510f3dad42eb800f891f55f8317f03c4bd0b21b6243ed1f4a000"
Sep 17 16:53:38.912941 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:38.912920    2581 scope.go:117] "RemoveContainer" containerID="f929fd62f596510f3dad42eb800f891f55f8317f03c4bd0b21b6243ed1f4a000"
Sep 17 16:53:38.913263 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:53:38.913246    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f929fd62f596510f3dad42eb800f891f55f8317f03c4bd0b21b6243ed1f4a000\": container with ID starting with f929fd62f596510f3dad42eb800f891f55f8317f03c4bd0b21b6243ed1f4a000 not found: ID does not exist" containerID="f929fd62f596510f3dad42eb800f891f55f8317f03c4bd0b21b6243ed1f4a000"
Sep 17 16:53:38.913319 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:38.913272    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f929fd62f596510f3dad42eb800f891f55f8317f03c4bd0b21b6243ed1f4a000"} err="failed to get container status \"f929fd62f596510f3dad42eb800f891f55f8317f03c4bd0b21b6243ed1f4a000\": rpc error: code = NotFound desc = could not find container \"f929fd62f596510f3dad42eb800f891f55f8317f03c4bd0b21b6243ed1f4a000\": container with ID starting with f929fd62f596510f3dad42eb800f891f55f8317f03c4bd0b21b6243ed1f4a000 not found: ID does not exist"
Sep 17 16:53:38.922262 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:38.922235    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8p9sc"]
Sep 17 16:53:38.928324 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:38.928303    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-8p9sc"]
Sep 17 16:53:40.835760 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:40.835705    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="415a2b92-8c2a-4aa3-b925-cc125557c202" path="/var/lib/kubelet/pods/415a2b92-8c2a-4aa3-b925-cc125557c202/volumes"
Sep 17 16:53:57.767729 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:57.767640    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dfs95"]
Sep 17 16:53:57.768131 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:57.767863    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-dfs95" podUID="2dae6917-b2a3-46c1-9505-03fd253c93a5" containerName="authorino" containerID="cri-o://06f836a540bba621888e96614abeeea3f2c2a0c8fe028c0f531b582f163bb2b9" gracePeriod=30
Sep 17 16:53:57.993514 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:57.993373    2581 generic.go:356] "Generic (PLEG): container finished" podID="2dae6917-b2a3-46c1-9505-03fd253c93a5" containerID="06f836a540bba621888e96614abeeea3f2c2a0c8fe028c0f531b582f163bb2b9" exitCode=0
Sep 17 16:53:57.993514 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:57.993437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dfs95" event={"ID":"2dae6917-b2a3-46c1-9505-03fd253c93a5","Type":"ContainerDied","Data":"06f836a540bba621888e96614abeeea3f2c2a0c8fe028c0f531b582f163bb2b9"}
Sep 17 16:53:58.006467 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:58.006448    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dfs95"
Sep 17 16:53:58.109844 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:58.109790    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdkjk\" (UniqueName: \"kubernetes.io/projected/2dae6917-b2a3-46c1-9505-03fd253c93a5-kube-api-access-fdkjk\") pod \"2dae6917-b2a3-46c1-9505-03fd253c93a5\" (UID: \"2dae6917-b2a3-46c1-9505-03fd253c93a5\") "
Sep 17 16:53:58.111749 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:58.111725    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dae6917-b2a3-46c1-9505-03fd253c93a5-kube-api-access-fdkjk" (OuterVolumeSpecName: "kube-api-access-fdkjk") pod "2dae6917-b2a3-46c1-9505-03fd253c93a5" (UID: "2dae6917-b2a3-46c1-9505-03fd253c93a5"). InnerVolumeSpecName "kube-api-access-fdkjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:53:58.210790 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:58.210765    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdkjk\" (UniqueName: \"kubernetes.io/projected/2dae6917-b2a3-46c1-9505-03fd253c93a5-kube-api-access-fdkjk\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:53:58.998942 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:58.998912    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dfs95"
Sep 17 16:53:58.999347 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:58.998960    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dfs95" event={"ID":"2dae6917-b2a3-46c1-9505-03fd253c93a5","Type":"ContainerDied","Data":"0c47c6c5af2ebc9231a4b1844ef838fe604c611caf2b9f6a5cb82d8e32da8a82"}
Sep 17 16:53:58.999347 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:58.999029    2581 scope.go:117] "RemoveContainer" containerID="06f836a540bba621888e96614abeeea3f2c2a0c8fe028c0f531b582f163bb2b9"
Sep 17 16:53:59.020207 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:59.020181    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dfs95"]
Sep 17 16:53:59.027008 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:53:59.026948    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dfs95"]
Sep 17 16:54:00.835646 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:54:00.835614    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dae6917-b2a3-46c1-9505-03fd253c93a5" path="/var/lib/kubelet/pods/2dae6917-b2a3-46c1-9505-03fd253c93a5/volumes"
Sep 17 16:57:02.761543 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:02.761466    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 16:57:02.765070 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:02.765047    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 16:57:52.474388 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.474352    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"]
Sep 17 16:57:52.475872 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.474714    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dae6917-b2a3-46c1-9505-03fd253c93a5" containerName="authorino"
Sep 17 16:57:52.475872 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.474727    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dae6917-b2a3-46c1-9505-03fd253c93a5" containerName="authorino"
Sep 17 16:57:52.475872 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.474740    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="415a2b92-8c2a-4aa3-b925-cc125557c202" containerName="authorino"
Sep 17 16:57:52.475872 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.474745    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="415a2b92-8c2a-4aa3-b925-cc125557c202" containerName="authorino"
Sep 17 16:57:52.475872 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.474816    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dae6917-b2a3-46c1-9505-03fd253c93a5" containerName="authorino"
Sep 17 16:57:52.475872 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.474828    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="415a2b92-8c2a-4aa3-b925-cc125557c202" containerName="authorino"
Sep 17 16:57:52.476854 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.476826    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"
Sep 17 16:57:52.480353 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.480335    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 16:57:52.480479 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.480332    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 16:57:52.480479 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.480332    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-vmnb8\""
Sep 17 16:57:52.480479 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.480398    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-rhdkc\""
Sep 17 16:57:52.480479 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.480337    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 16:57:52.490853 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.490831    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"]
Sep 17 16:57:52.513196 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.513168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b7d8a3-c8f2-4263-9d0a-9db2ac62de16-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747\" (UID: \"94b7d8a3-c8f2-4263-9d0a-9db2ac62de16\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"
Sep 17 16:57:52.513330 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.513199    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77gfq\" (UniqueName: \"kubernetes.io/projected/94b7d8a3-c8f2-4263-9d0a-9db2ac62de16-kube-api-access-77gfq\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747\" (UID: \"94b7d8a3-c8f2-4263-9d0a-9db2ac62de16\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"
Sep 17 16:57:52.613918 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.613893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77gfq\" (UniqueName: \"kubernetes.io/projected/94b7d8a3-c8f2-4263-9d0a-9db2ac62de16-kube-api-access-77gfq\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747\" (UID: \"94b7d8a3-c8f2-4263-9d0a-9db2ac62de16\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"
Sep 17 16:57:52.614041 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.614008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b7d8a3-c8f2-4263-9d0a-9db2ac62de16-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747\" (UID: \"94b7d8a3-c8f2-4263-9d0a-9db2ac62de16\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"
Sep 17 16:57:52.616337 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.616319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b7d8a3-c8f2-4263-9d0a-9db2ac62de16-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747\" (UID: \"94b7d8a3-c8f2-4263-9d0a-9db2ac62de16\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"
Sep 17 16:57:52.624667 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.624649    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77gfq\" (UniqueName: \"kubernetes.io/projected/94b7d8a3-c8f2-4263-9d0a-9db2ac62de16-kube-api-access-77gfq\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747\" (UID: \"94b7d8a3-c8f2-4263-9d0a-9db2ac62de16\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"
Sep 17 16:57:52.787190 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:52.787133    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"
Sep 17 16:57:53.090752 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:53.090725    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"]
Sep 17 16:57:53.091409 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:57:53.091381    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b7d8a3_c8f2_4263_9d0a_9db2ac62de16.slice/crio-7c603bc7eacccd546167c92eb5dc461f9014d087483d8d66730401f604359653 WatchSource:0}: Error finding container 7c603bc7eacccd546167c92eb5dc461f9014d087483d8d66730401f604359653: Status 404 returned error can't find the container with id 7c603bc7eacccd546167c92eb5dc461f9014d087483d8d66730401f604359653
Sep 17 16:57:53.093310 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:53.093292    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:57:53.109400 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:53.109365    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747" event={"ID":"94b7d8a3-c8f2-4263-9d0a-9db2ac62de16","Type":"ContainerStarted","Data":"7c603bc7eacccd546167c92eb5dc461f9014d087483d8d66730401f604359653"}
Sep 17 16:57:56.125178 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.125140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747" event={"ID":"94b7d8a3-c8f2-4263-9d0a-9db2ac62de16","Type":"ContainerStarted","Data":"90751d6b366fc4ca4be45153d86639abb52a7bee29b07a7c3f56f26c6a5c720f"}
Sep 17 16:57:56.125592 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.125309    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"
Sep 17 16:57:56.147289 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.147233    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747" podStartSLOduration=1.332795763 podStartE2EDuration="4.147214162s" totalImagesPullingTime="2.814418399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:57:56.144397436 +0000 UTC m=+955.567335299" watchObservedRunningTime="2026-09-17 16:57:56.147214162 +0000 UTC m=+955.570152026"
Sep 17 16:57:56.691269 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.691236    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"]
Sep 17 16:57:56.693768 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.693746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.696313 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.696288    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 16:57:56.696313 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.696312    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-bf47f\""
Sep 17 16:57:56.697226 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.697205    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 16:57:56.697297 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.697219    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 16:57:56.712528 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.712507    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"]
Sep 17 16:57:56.746091 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.746070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cj5g\" (UniqueName: \"kubernetes.io/projected/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-kube-api-access-5cj5g\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.746195 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.746101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.746195 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.746128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.746195 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.746154    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.746314 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.746195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.746314 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.746259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.847481 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.847456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.847576 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.847487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.847576 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.847503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.847576 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.847528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.847729 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.847594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cj5g\" (UniqueName: \"kubernetes.io/projected/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-kube-api-access-5cj5g\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.847729 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.847626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.847897 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.847872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.847962 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.847894    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.848041 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.847997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.849823 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.849802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.850011 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.849983    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:56.858034 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:56.858011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cj5g\" (UniqueName: \"kubernetes.io/projected/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-kube-api-access-5cj5g\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:57.004619 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:57.004559    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:57:57.281670 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:57.281629    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"]
Sep 17 16:57:57.283708 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:57:57.283683    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda069a53f_c25e_48a5_9efd_fbe0c4a534f9.slice/crio-00a73afce8ef06e6dfc87fb612d647d89766b65b0381e689154feb7f41a0c437 WatchSource:0}: Error finding container 00a73afce8ef06e6dfc87fb612d647d89766b65b0381e689154feb7f41a0c437: Status 404 returned error can't find the container with id 00a73afce8ef06e6dfc87fb612d647d89766b65b0381e689154feb7f41a0c437
Sep 17 16:57:58.138257 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:58.138219    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt" event={"ID":"a069a53f-c25e-48a5-9efd-fbe0c4a534f9","Type":"ContainerStarted","Data":"00a73afce8ef06e6dfc87fb612d647d89766b65b0381e689154feb7f41a0c437"}
Sep 17 16:57:59.145229 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:59.145198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt" event={"ID":"a069a53f-c25e-48a5-9efd-fbe0c4a534f9","Type":"ContainerStarted","Data":"f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4"}
Sep 17 16:57:59.171468 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:57:59.171417    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt" podStartSLOduration=1.955479617 podStartE2EDuration="3.171401255s" totalImagesPullingTime="1.215921638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:57:59.169366915 +0000 UTC m=+958.592304769" watchObservedRunningTime="2026-09-17 16:57:59.171401255 +0000 UTC m=+958.594339155"
Sep 17 16:58:00.005685 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:00.005656    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:58:07.005378 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:07.005335    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:58:07.018232 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:07.018199    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:58:07.194282 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:07.194253    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:58:27.132286 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:27.132195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"
Sep 17 16:58:33.173233 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.173197    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"]
Sep 17 16:58:33.184466 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:58:33.184426    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:58:33.184466 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:58:33.184466 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 16:58:33.184466 ip-10-0-140-49 kubenswrapper[2581]:  > containerID="f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4" cmd=["/bin/sleep","15"]
Sep 17 16:58:33.184735 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.184492    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt" message="" err=<
Sep 17 16:58:33.184735 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:58:33.184735 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 16:58:33.184735 ip-10-0-140-49 kubenswrapper[2581]:  >
Sep 17 16:58:33.184735 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:58:33.184509    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:58:33.184735 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:58:33.184735 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 16:58:33.184735 ip-10-0-140-49 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt" podUID="a069a53f-c25e-48a5-9efd-fbe0c4a534f9" containerName="main" containerID="cri-o://f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4"
Sep 17 16:58:33.184735 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.184562    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt" podUID="a069a53f-c25e-48a5-9efd-fbe0c4a534f9" containerName="main" containerID="cri-o://f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4" gracePeriod=60
Sep 17 16:58:33.527504 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.527481    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:58:33.637751 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.637722    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-model-cache\") pod \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") "
Sep 17 16:58:33.637751 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.637751    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-tmp-dir\") pod \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") "
Sep 17 16:58:33.637936 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.637805    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cj5g\" (UniqueName: \"kubernetes.io/projected/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-kube-api-access-5cj5g\") pod \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") "
Sep 17 16:58:33.637936 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.637851    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-tls-certs\") pod \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") "
Sep 17 16:58:33.637936 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.637888    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-dshm\") pod \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") "
Sep 17 16:58:33.637936 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.637917    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-home\") pod \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\" (UID: \"a069a53f-c25e-48a5-9efd-fbe0c4a534f9\") "
Sep 17 16:58:33.638167 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.637930    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a069a53f-c25e-48a5-9efd-fbe0c4a534f9" (UID: "a069a53f-c25e-48a5-9efd-fbe0c4a534f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:58:33.638167 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.638044    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a069a53f-c25e-48a5-9efd-fbe0c4a534f9" (UID: "a069a53f-c25e-48a5-9efd-fbe0c4a534f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:58:33.638278 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.638169    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-home" (OuterVolumeSpecName: "home") pod "a069a53f-c25e-48a5-9efd-fbe0c4a534f9" (UID: "a069a53f-c25e-48a5-9efd-fbe0c4a534f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:58:33.638278 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.638208    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:58:33.638278 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.638228    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:58:33.640093 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.640062    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-dshm" (OuterVolumeSpecName: "dshm") pod "a069a53f-c25e-48a5-9efd-fbe0c4a534f9" (UID: "a069a53f-c25e-48a5-9efd-fbe0c4a534f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:58:33.640200 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.640112    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a069a53f-c25e-48a5-9efd-fbe0c4a534f9" (UID: "a069a53f-c25e-48a5-9efd-fbe0c4a534f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:58:33.640247 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.640220    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-kube-api-access-5cj5g" (OuterVolumeSpecName: "kube-api-access-5cj5g") pod "a069a53f-c25e-48a5-9efd-fbe0c4a534f9" (UID: "a069a53f-c25e-48a5-9efd-fbe0c4a534f9"). InnerVolumeSpecName "kube-api-access-5cj5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:58:33.739056 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.738997    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cj5g\" (UniqueName: \"kubernetes.io/projected/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-kube-api-access-5cj5g\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:58:33.739056 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.739021    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:58:33.739056 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.739035    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:58:33.739056 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:33.739045    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a069a53f-c25e-48a5-9efd-fbe0c4a534f9-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:58:34.309319 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:34.309284    2581 generic.go:356] "Generic (PLEG): container finished" podID="a069a53f-c25e-48a5-9efd-fbe0c4a534f9" containerID="f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4" exitCode=0
Sep 17 16:58:34.309706 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:34.309333    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"
Sep 17 16:58:34.309706 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:34.309377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt" event={"ID":"a069a53f-c25e-48a5-9efd-fbe0c4a534f9","Type":"ContainerDied","Data":"f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4"}
Sep 17 16:58:34.309706 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:34.309428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt" event={"ID":"a069a53f-c25e-48a5-9efd-fbe0c4a534f9","Type":"ContainerDied","Data":"00a73afce8ef06e6dfc87fb612d647d89766b65b0381e689154feb7f41a0c437"}
Sep 17 16:58:34.309706 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:34.309451    2581 scope.go:117] "RemoveContainer" containerID="f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4"
Sep 17 16:58:34.318551 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:34.318535    2581 scope.go:117] "RemoveContainer" containerID="f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4"
Sep 17 16:58:34.318805 ip-10-0-140-49 kubenswrapper[2581]: E0917 16:58:34.318785    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4\": container with ID starting with f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4 not found: ID does not exist" containerID="f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4"
Sep 17 16:58:34.318861 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:34.318811    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4"} err="failed to get container status \"f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4\": rpc error: code = NotFound desc = could not find container \"f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4\": container with ID starting with f6b19a5e74e3b36ec4852baafa49e983ad26b2e4e78966d76d4c4e077f5ae3c4 not found: ID does not exist"
Sep 17 16:58:34.358719 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:34.358697    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"]
Sep 17 16:58:34.369144 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:34.369118    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mwrgt"]
Sep 17 16:58:34.835438 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:34.835412    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a069a53f-c25e-48a5-9efd-fbe0c4a534f9" path="/var/lib/kubelet/pods/a069a53f-c25e-48a5-9efd-fbe0c4a534f9/volumes"
Sep 17 16:58:35.342389 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:35.342356    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"]
Sep 17 16:58:37.132344 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:37.132297    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747" podUID="94b7d8a3-c8f2-4263-9d0a-9db2ac62de16" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:58:47.132024 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:47.131952    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747" podUID="94b7d8a3-c8f2-4263-9d0a-9db2ac62de16" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:58:50.361860 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:50.361813    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747" podUID="94b7d8a3-c8f2-4263-9d0a-9db2ac62de16" containerName="main" containerID="cri-o://90751d6b366fc4ca4be45153d86639abb52a7bee29b07a7c3f56f26c6a5c720f" gracePeriod=45
Sep 17 16:58:57.132857 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:57.132815    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747" podUID="94b7d8a3-c8f2-4263-9d0a-9db2ac62de16" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:58:57.133320 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:58:57.132920    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"
Sep 17 16:59:07.132553 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:07.132508    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747" podUID="94b7d8a3-c8f2-4263-9d0a-9db2ac62de16" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:59:17.132448 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:17.132401    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747" podUID="94b7d8a3-c8f2-4263-9d0a-9db2ac62de16" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:59:20.523955 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:20.523920    2581 generic.go:356] "Generic (PLEG): container finished" podID="94b7d8a3-c8f2-4263-9d0a-9db2ac62de16" containerID="90751d6b366fc4ca4be45153d86639abb52a7bee29b07a7c3f56f26c6a5c720f" exitCode=0
Sep 17 16:59:20.524304 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:20.524005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747" event={"ID":"94b7d8a3-c8f2-4263-9d0a-9db2ac62de16","Type":"ContainerDied","Data":"90751d6b366fc4ca4be45153d86639abb52a7bee29b07a7c3f56f26c6a5c720f"}
Sep 17 16:59:20.608408 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:20.608382    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"
Sep 17 16:59:20.712327 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:20.712303    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b7d8a3-c8f2-4263-9d0a-9db2ac62de16-tls-certs\") pod \"94b7d8a3-c8f2-4263-9d0a-9db2ac62de16\" (UID: \"94b7d8a3-c8f2-4263-9d0a-9db2ac62de16\") "
Sep 17 16:59:20.712439 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:20.712372    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77gfq\" (UniqueName: \"kubernetes.io/projected/94b7d8a3-c8f2-4263-9d0a-9db2ac62de16-kube-api-access-77gfq\") pod \"94b7d8a3-c8f2-4263-9d0a-9db2ac62de16\" (UID: \"94b7d8a3-c8f2-4263-9d0a-9db2ac62de16\") "
Sep 17 16:59:20.714377 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:20.714344    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b7d8a3-c8f2-4263-9d0a-9db2ac62de16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94b7d8a3-c8f2-4263-9d0a-9db2ac62de16" (UID: "94b7d8a3-c8f2-4263-9d0a-9db2ac62de16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:59:20.714478 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:20.714387    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b7d8a3-c8f2-4263-9d0a-9db2ac62de16-kube-api-access-77gfq" (OuterVolumeSpecName: "kube-api-access-77gfq") pod "94b7d8a3-c8f2-4263-9d0a-9db2ac62de16" (UID: "94b7d8a3-c8f2-4263-9d0a-9db2ac62de16"). InnerVolumeSpecName "kube-api-access-77gfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:59:20.813499 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:20.813476    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b7d8a3-c8f2-4263-9d0a-9db2ac62de16-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:59:20.813499 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:20.813500    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77gfq\" (UniqueName: \"kubernetes.io/projected/94b7d8a3-c8f2-4263-9d0a-9db2ac62de16-kube-api-access-77gfq\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 16:59:21.530019 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:21.529985    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747" event={"ID":"94b7d8a3-c8f2-4263-9d0a-9db2ac62de16","Type":"ContainerDied","Data":"7c603bc7eacccd546167c92eb5dc461f9014d087483d8d66730401f604359653"}
Sep 17 16:59:21.530466 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:21.530016    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"
Sep 17 16:59:21.530466 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:21.530057    2581 scope.go:117] "RemoveContainer" containerID="90751d6b366fc4ca4be45153d86639abb52a7bee29b07a7c3f56f26c6a5c720f"
Sep 17 16:59:21.549170 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:21.549142    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"]
Sep 17 16:59:21.553680 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:21.553656    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-n9747"]
Sep 17 16:59:22.834526 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:22.834496    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94b7d8a3-c8f2-4263-9d0a-9db2ac62de16" path="/var/lib/kubelet/pods/94b7d8a3-c8f2-4263-9d0a-9db2ac62de16/volumes"
Sep 17 16:59:33.303258 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.303227    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"]
Sep 17 16:59:33.303709 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.303563    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b7d8a3-c8f2-4263-9d0a-9db2ac62de16" containerName="main"
Sep 17 16:59:33.303709 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.303575    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b7d8a3-c8f2-4263-9d0a-9db2ac62de16" containerName="main"
Sep 17 16:59:33.303709 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.303602    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a069a53f-c25e-48a5-9efd-fbe0c4a534f9" containerName="main"
Sep 17 16:59:33.303709 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.303608    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a069a53f-c25e-48a5-9efd-fbe0c4a534f9" containerName="main"
Sep 17 16:59:33.303709 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.303659    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b7d8a3-c8f2-4263-9d0a-9db2ac62de16" containerName="main"
Sep 17 16:59:33.303709 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.303673    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a069a53f-c25e-48a5-9efd-fbe0c4a534f9" containerName="main"
Sep 17 16:59:33.307988 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.307955    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:33.311292 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.311268    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-vsj2w\""
Sep 17 16:59:33.311292 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.311286    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 16:59:33.311452 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.311435    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 16:59:33.311603 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.311582    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 16:59:33.311743 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.311651    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-4xkr9\""
Sep 17 16:59:33.318038 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.318019    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"]
Sep 17 16:59:33.406926 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.406902    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3021c9d3-6176-4d03-a5ed-d24f143925b6-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc94drc\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:33.407052 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.406949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3021c9d3-6176-4d03-a5ed-d24f143925b6-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc94drc\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:33.407052 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.406989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc9tg\" (UniqueName: \"kubernetes.io/projected/3021c9d3-6176-4d03-a5ed-d24f143925b6-kube-api-access-tc9tg\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc94drc\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:33.407129 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.407073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3021c9d3-6176-4d03-a5ed-d24f143925b6-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc94drc\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:33.507757 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.507731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3021c9d3-6176-4d03-a5ed-d24f143925b6-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc94drc\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:33.507853 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.507774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3021c9d3-6176-4d03-a5ed-d24f143925b6-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc94drc\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:33.507853 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.507826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3021c9d3-6176-4d03-a5ed-d24f143925b6-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc94drc\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:33.507956 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.507858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc9tg\" (UniqueName: \"kubernetes.io/projected/3021c9d3-6176-4d03-a5ed-d24f143925b6-kube-api-access-tc9tg\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc94drc\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:33.508174 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.508151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3021c9d3-6176-4d03-a5ed-d24f143925b6-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc94drc\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:33.508245 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.508195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3021c9d3-6176-4d03-a5ed-d24f143925b6-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc94drc\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:33.510232 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.510210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3021c9d3-6176-4d03-a5ed-d24f143925b6-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc94drc\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:33.517883 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.517863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc9tg\" (UniqueName: \"kubernetes.io/projected/3021c9d3-6176-4d03-a5ed-d24f143925b6-kube-api-access-tc9tg\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffc94drc\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:33.619444 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.619390    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:33.894238 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:33.894176    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"]
Sep 17 16:59:33.897531 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:59:33.897500    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3021c9d3_6176_4d03_a5ed_d24f143925b6.slice/crio-b130e20c4c6b08a5264e6b3b3a1c924aa45a5c9afb582ac4211b15f1da464028 WatchSource:0}: Error finding container b130e20c4c6b08a5264e6b3b3a1c924aa45a5c9afb582ac4211b15f1da464028: Status 404 returned error can't find the container with id b130e20c4c6b08a5264e6b3b3a1c924aa45a5c9afb582ac4211b15f1da464028
Sep 17 16:59:34.593544 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:34.593508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc" event={"ID":"3021c9d3-6176-4d03-a5ed-d24f143925b6","Type":"ContainerStarted","Data":"5d9d72eaadce322664fcdc1bd786d3a6c949af6d0e29a9ed341b8ea6a2cbb5e6"}
Sep 17 16:59:34.593987 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:34.593554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc" event={"ID":"3021c9d3-6176-4d03-a5ed-d24f143925b6","Type":"ContainerStarted","Data":"b130e20c4c6b08a5264e6b3b3a1c924aa45a5c9afb582ac4211b15f1da464028"}
Sep 17 16:59:34.593987 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:34.593631    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 16:59:34.613371 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:34.613324    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc" podStartSLOduration=1.613308725 podStartE2EDuration="1.613308725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:59:34.61222112 +0000 UTC m=+1054.035159005" watchObservedRunningTime="2026-09-17 16:59:34.613308725 +0000 UTC m=+1054.036246587"
Sep 17 16:59:47.079909 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.079826    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"]
Sep 17 16:59:47.083374 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.083351    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.085791 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.085766    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 16:59:47.086662 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.086638    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-8hmbd\""
Sep 17 16:59:47.086758 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.086683    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 16:59:47.086758 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.086701    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 16:59:47.100573 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.100548    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"]
Sep 17 16:59:47.212216 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.212188    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0000023b-704b-4b7f-b80d-3609143a0973-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.212312 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.212243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l49lc\" (UniqueName: \"kubernetes.io/projected/0000023b-704b-4b7f-b80d-3609143a0973-kube-api-access-l49lc\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.212312 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.212267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.212312 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.212284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.212312 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.212304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-home\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.212448 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.212402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.313644 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.313619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.313750 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.313658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0000023b-704b-4b7f-b80d-3609143a0973-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.313750 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.313692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l49lc\" (UniqueName: \"kubernetes.io/projected/0000023b-704b-4b7f-b80d-3609143a0973-kube-api-access-l49lc\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.313750 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.313713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.313750 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.313737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.313960 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.313757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-home\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.314152 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.314130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.314216 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.314147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-home\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.314216 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.314180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.315872 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.315855    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.316275 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.316256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0000023b-704b-4b7f-b80d-3609143a0973-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.328879 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.328853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l49lc\" (UniqueName: \"kubernetes.io/projected/0000023b-704b-4b7f-b80d-3609143a0973-kube-api-access-l49lc\") pod \"fc-concurrency-test-kserve-b5d44467c-cqpzc\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.394667 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.394610    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:47.670260 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:47.670239    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"]
Sep 17 16:59:47.672204 ip-10-0-140-49 kubenswrapper[2581]: W0917 16:59:47.672179    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0000023b_704b_4b7f_b80d_3609143a0973.slice/crio-cc8b1d615049bee857bf4dfb257a7d09c0f7235086b54b50b397e5386f073d9c WatchSource:0}: Error finding container cc8b1d615049bee857bf4dfb257a7d09c0f7235086b54b50b397e5386f073d9c: Status 404 returned error can't find the container with id cc8b1d615049bee857bf4dfb257a7d09c0f7235086b54b50b397e5386f073d9c
Sep 17 16:59:48.664771 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:48.664735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc" event={"ID":"0000023b-704b-4b7f-b80d-3609143a0973","Type":"ContainerStarted","Data":"7fbffa00c0fbc653160c145eb3c1b3f12f37989a620ccabf8cb58cb582b32feb"}
Sep 17 16:59:48.665235 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:48.664784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc" event={"ID":"0000023b-704b-4b7f-b80d-3609143a0973","Type":"ContainerStarted","Data":"cc8b1d615049bee857bf4dfb257a7d09c0f7235086b54b50b397e5386f073d9c"}
Sep 17 16:59:48.688085 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:48.688041    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc" podStartSLOduration=1.688017917 podStartE2EDuration="1.688017917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:59:48.687215508 +0000 UTC m=+1068.110153393" watchObservedRunningTime="2026-09-17 16:59:48.688017917 +0000 UTC m=+1068.110955777"
Sep 17 16:59:49.394760 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:49.394727    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:57.395156 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:57.395117    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:57.407318 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:57.407284    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 16:59:57.719031 ip-10-0-140-49 kubenswrapper[2581]: I0917 16:59:57.718998    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 17:00:05.599641 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:05.599611    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 17:00:11.536214 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:11.536171    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"]
Sep 17 17:00:15.599531 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:15.599493    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc" podUID="3021c9d3-6176-4d03-a5ed-d24f143925b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:00:25.599052 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:25.599009    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc" podUID="3021c9d3-6176-4d03-a5ed-d24f143925b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:00:26.466304 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:26.466270    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"]
Sep 17 17:00:26.475595 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:00:26.475560    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:00:26.475595 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:00:26.475595 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:00:26.475595 ip-10-0-140-49 kubenswrapper[2581]:  > containerID="7fbffa00c0fbc653160c145eb3c1b3f12f37989a620ccabf8cb58cb582b32feb" cmd=["/bin/sleep","15"]
Sep 17 17:00:26.475900 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:26.475609    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc" message="" err=<
Sep 17 17:00:26.475900 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:00:26.475900 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:00:26.475900 ip-10-0-140-49 kubenswrapper[2581]:  >
Sep 17 17:00:26.475900 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:00:26.475621    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:00:26.475900 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:00:26.475900 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:00:26.475900 ip-10-0-140-49 kubenswrapper[2581]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc" podUID="0000023b-704b-4b7f-b80d-3609143a0973" containerName="main" containerID="cri-o://7fbffa00c0fbc653160c145eb3c1b3f12f37989a620ccabf8cb58cb582b32feb"
Sep 17 17:00:26.475900 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:26.475664    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc" podUID="0000023b-704b-4b7f-b80d-3609143a0973" containerName="main" containerID="cri-o://7fbffa00c0fbc653160c145eb3c1b3f12f37989a620ccabf8cb58cb582b32feb" gracePeriod=60
Sep 17 17:00:26.555012 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:26.554950    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc" podUID="3021c9d3-6176-4d03-a5ed-d24f143925b6" containerName="main" containerID="cri-o://5d9d72eaadce322664fcdc1bd786d3a6c949af6d0e29a9ed341b8ea6a2cbb5e6" gracePeriod=45
Sep 17 17:00:26.708216 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:26.708177    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc" podUID="0000023b-704b-4b7f-b80d-3609143a0973" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 17 17:00:26.848503 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:26.848419    2581 generic.go:356] "Generic (PLEG): container finished" podID="0000023b-704b-4b7f-b80d-3609143a0973" containerID="7fbffa00c0fbc653160c145eb3c1b3f12f37989a620ccabf8cb58cb582b32feb" exitCode=0
Sep 17 17:00:26.848503 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:26.848482    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc" event={"ID":"0000023b-704b-4b7f-b80d-3609143a0973","Type":"ContainerDied","Data":"7fbffa00c0fbc653160c145eb3c1b3f12f37989a620ccabf8cb58cb582b32feb"}
Sep 17 17:00:27.215535 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.215510    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 17:00:27.349936 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.349906    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l49lc\" (UniqueName: \"kubernetes.io/projected/0000023b-704b-4b7f-b80d-3609143a0973-kube-api-access-l49lc\") pod \"0000023b-704b-4b7f-b80d-3609143a0973\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") "
Sep 17 17:00:27.350110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.349993    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-dshm\") pod \"0000023b-704b-4b7f-b80d-3609143a0973\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") "
Sep 17 17:00:27.350110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.350024    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0000023b-704b-4b7f-b80d-3609143a0973-tls-certs\") pod \"0000023b-704b-4b7f-b80d-3609143a0973\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") "
Sep 17 17:00:27.350110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.350055    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-model-cache\") pod \"0000023b-704b-4b7f-b80d-3609143a0973\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") "
Sep 17 17:00:27.350110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.350107    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-tmp-dir\") pod \"0000023b-704b-4b7f-b80d-3609143a0973\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") "
Sep 17 17:00:27.350261 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.350124    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-home\") pod \"0000023b-704b-4b7f-b80d-3609143a0973\" (UID: \"0000023b-704b-4b7f-b80d-3609143a0973\") "
Sep 17 17:00:27.350416 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.350385    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-model-cache" (OuterVolumeSpecName: "model-cache") pod "0000023b-704b-4b7f-b80d-3609143a0973" (UID: "0000023b-704b-4b7f-b80d-3609143a0973"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:00:27.350513 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.350401    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0000023b-704b-4b7f-b80d-3609143a0973" (UID: "0000023b-704b-4b7f-b80d-3609143a0973"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:00:27.350513 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.350465    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-home" (OuterVolumeSpecName: "home") pod "0000023b-704b-4b7f-b80d-3609143a0973" (UID: "0000023b-704b-4b7f-b80d-3609143a0973"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:00:27.352270 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.352238    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0000023b-704b-4b7f-b80d-3609143a0973-kube-api-access-l49lc" (OuterVolumeSpecName: "kube-api-access-l49lc") pod "0000023b-704b-4b7f-b80d-3609143a0973" (UID: "0000023b-704b-4b7f-b80d-3609143a0973"). InnerVolumeSpecName "kube-api-access-l49lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:00:27.352368 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.352291    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0000023b-704b-4b7f-b80d-3609143a0973-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0000023b-704b-4b7f-b80d-3609143a0973" (UID: "0000023b-704b-4b7f-b80d-3609143a0973"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:00:27.352368 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.352307    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-dshm" (OuterVolumeSpecName: "dshm") pod "0000023b-704b-4b7f-b80d-3609143a0973" (UID: "0000023b-704b-4b7f-b80d-3609143a0973"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:00:27.450696 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.450669    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l49lc\" (UniqueName: \"kubernetes.io/projected/0000023b-704b-4b7f-b80d-3609143a0973-kube-api-access-l49lc\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:00:27.450696 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.450694    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:00:27.450870 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.450703    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0000023b-704b-4b7f-b80d-3609143a0973-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:00:27.450870 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.450713    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:00:27.450870 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.450721    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:00:27.450870 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.450728    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0000023b-704b-4b7f-b80d-3609143a0973-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:00:27.858201 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.858128    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"
Sep 17 17:00:27.858201 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.858138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc" event={"ID":"0000023b-704b-4b7f-b80d-3609143a0973","Type":"ContainerDied","Data":"cc8b1d615049bee857bf4dfb257a7d09c0f7235086b54b50b397e5386f073d9c"}
Sep 17 17:00:27.858201 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.858199    2581 scope.go:117] "RemoveContainer" containerID="7fbffa00c0fbc653160c145eb3c1b3f12f37989a620ccabf8cb58cb582b32feb"
Sep 17 17:00:27.883679 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.883657    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"]
Sep 17 17:00:27.886140 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:27.886120    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-cqpzc"]
Sep 17 17:00:28.836128 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:28.836096    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0000023b-704b-4b7f-b80d-3609143a0973" path="/var/lib/kubelet/pods/0000023b-704b-4b7f-b80d-3609143a0973/volumes"
Sep 17 17:00:35.599518 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:35.599483    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc" podUID="3021c9d3-6176-4d03-a5ed-d24f143925b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:00:35.599985 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:35.599598    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 17:00:45.599744 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:45.599705    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc" podUID="3021c9d3-6176-4d03-a5ed-d24f143925b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:00:55.599781 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:55.599740    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc" podUID="3021c9d3-6176-4d03-a5ed-d24f143925b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:00:56.803814 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:56.803791    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 17:00:56.906229 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:56.906159    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3021c9d3-6176-4d03-a5ed-d24f143925b6-cachi2-cache\") pod \"3021c9d3-6176-4d03-a5ed-d24f143925b6\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") "
Sep 17 17:00:56.906229 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:56.906190    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc9tg\" (UniqueName: \"kubernetes.io/projected/3021c9d3-6176-4d03-a5ed-d24f143925b6-kube-api-access-tc9tg\") pod \"3021c9d3-6176-4d03-a5ed-d24f143925b6\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") "
Sep 17 17:00:56.906229 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:56.906225    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3021c9d3-6176-4d03-a5ed-d24f143925b6-tls-certs\") pod \"3021c9d3-6176-4d03-a5ed-d24f143925b6\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") "
Sep 17 17:00:56.906470 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:56.906264    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3021c9d3-6176-4d03-a5ed-d24f143925b6-tokenizer-cache\") pod \"3021c9d3-6176-4d03-a5ed-d24f143925b6\" (UID: \"3021c9d3-6176-4d03-a5ed-d24f143925b6\") "
Sep 17 17:00:56.906527 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:56.906466    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3021c9d3-6176-4d03-a5ed-d24f143925b6-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "3021c9d3-6176-4d03-a5ed-d24f143925b6" (UID: "3021c9d3-6176-4d03-a5ed-d24f143925b6"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:00:56.906596 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:56.906574    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3021c9d3-6176-4d03-a5ed-d24f143925b6-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "3021c9d3-6176-4d03-a5ed-d24f143925b6" (UID: "3021c9d3-6176-4d03-a5ed-d24f143925b6"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:00:56.908323 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:56.908304    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3021c9d3-6176-4d03-a5ed-d24f143925b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3021c9d3-6176-4d03-a5ed-d24f143925b6" (UID: "3021c9d3-6176-4d03-a5ed-d24f143925b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:00:56.908395 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:56.908377    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3021c9d3-6176-4d03-a5ed-d24f143925b6-kube-api-access-tc9tg" (OuterVolumeSpecName: "kube-api-access-tc9tg") pod "3021c9d3-6176-4d03-a5ed-d24f143925b6" (UID: "3021c9d3-6176-4d03-a5ed-d24f143925b6"). InnerVolumeSpecName "kube-api-access-tc9tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:00:56.994506 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:56.994477    2581 generic.go:356] "Generic (PLEG): container finished" podID="3021c9d3-6176-4d03-a5ed-d24f143925b6" containerID="5d9d72eaadce322664fcdc1bd786d3a6c949af6d0e29a9ed341b8ea6a2cbb5e6" exitCode=0
Sep 17 17:00:56.994621 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:56.994527    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"
Sep 17 17:00:56.994677 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:56.994530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc" event={"ID":"3021c9d3-6176-4d03-a5ed-d24f143925b6","Type":"ContainerDied","Data":"5d9d72eaadce322664fcdc1bd786d3a6c949af6d0e29a9ed341b8ea6a2cbb5e6"}
Sep 17 17:00:56.994725 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:56.994673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc" event={"ID":"3021c9d3-6176-4d03-a5ed-d24f143925b6","Type":"ContainerDied","Data":"b130e20c4c6b08a5264e6b3b3a1c924aa45a5c9afb582ac4211b15f1da464028"}
Sep 17 17:00:56.994725 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:56.994707    2581 scope.go:117] "RemoveContainer" containerID="5d9d72eaadce322664fcdc1bd786d3a6c949af6d0e29a9ed341b8ea6a2cbb5e6"
Sep 17 17:00:57.003297 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:57.003270    2581 scope.go:117] "RemoveContainer" containerID="5d9d72eaadce322664fcdc1bd786d3a6c949af6d0e29a9ed341b8ea6a2cbb5e6"
Sep 17 17:00:57.003559 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:00:57.003537    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9d72eaadce322664fcdc1bd786d3a6c949af6d0e29a9ed341b8ea6a2cbb5e6\": container with ID starting with 5d9d72eaadce322664fcdc1bd786d3a6c949af6d0e29a9ed341b8ea6a2cbb5e6 not found: ID does not exist" containerID="5d9d72eaadce322664fcdc1bd786d3a6c949af6d0e29a9ed341b8ea6a2cbb5e6"
Sep 17 17:00:57.003627 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:57.003566    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9d72eaadce322664fcdc1bd786d3a6c949af6d0e29a9ed341b8ea6a2cbb5e6"} err="failed to get container status \"5d9d72eaadce322664fcdc1bd786d3a6c949af6d0e29a9ed341b8ea6a2cbb5e6\": rpc error: code = NotFound desc = could not find container \"5d9d72eaadce322664fcdc1bd786d3a6c949af6d0e29a9ed341b8ea6a2cbb5e6\": container with ID starting with 5d9d72eaadce322664fcdc1bd786d3a6c949af6d0e29a9ed341b8ea6a2cbb5e6 not found: ID does not exist"
Sep 17 17:00:57.006934 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:57.006916    2581 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3021c9d3-6176-4d03-a5ed-d24f143925b6-cachi2-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:00:57.007033 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:57.006935    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc9tg\" (UniqueName: \"kubernetes.io/projected/3021c9d3-6176-4d03-a5ed-d24f143925b6-kube-api-access-tc9tg\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:00:57.007033 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:57.006946    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3021c9d3-6176-4d03-a5ed-d24f143925b6-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:00:57.007033 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:57.006956    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3021c9d3-6176-4d03-a5ed-d24f143925b6-tokenizer-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:00:57.017286 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:57.017265    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"]
Sep 17 17:00:57.019639 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:57.019620    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffc94drc"]
Sep 17 17:00:58.836264 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:00:58.836231    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3021c9d3-6176-4d03-a5ed-d24f143925b6" path="/var/lib/kubelet/pods/3021c9d3-6176-4d03-a5ed-d24f143925b6/volumes"
Sep 17 17:01:04.216182 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.216144    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"]
Sep 17 17:01:04.216575 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.216475    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0000023b-704b-4b7f-b80d-3609143a0973" containerName="main"
Sep 17 17:01:04.216575 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.216487    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0000023b-704b-4b7f-b80d-3609143a0973" containerName="main"
Sep 17 17:01:04.216575 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.216506    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3021c9d3-6176-4d03-a5ed-d24f143925b6" containerName="main"
Sep 17 17:01:04.216575 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.216512    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3021c9d3-6176-4d03-a5ed-d24f143925b6" containerName="main"
Sep 17 17:01:04.216575 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.216576    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3021c9d3-6176-4d03-a5ed-d24f143925b6" containerName="main"
Sep 17 17:01:04.216768 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.216583    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0000023b-704b-4b7f-b80d-3609143a0973" containerName="main"
Sep 17 17:01:04.221203 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.221181    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.224902 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.224879    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 17:01:04.225046 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.224945    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-75xt4\""
Sep 17 17:01:04.225843 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.225823    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 17:01:04.225936 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.225825    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 17:01:04.225936 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.225825    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-nrxmc\""
Sep 17 17:01:04.243058 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.243026    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"]
Sep 17 17:01:04.259920 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.259895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.260037 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.259940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5df99\" (UniqueName: \"kubernetes.io/projected/523f34e0-14f8-493b-ad56-20382baa47be-kube-api-access-5df99\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.260037 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.260008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.260037 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.260025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.260160 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.260046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.260160 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.260100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523f34e0-14f8-493b-ad56-20382baa47be-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.361148 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.361110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.361268 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.361183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5df99\" (UniqueName: \"kubernetes.io/projected/523f34e0-14f8-493b-ad56-20382baa47be-kube-api-access-5df99\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.361268 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.361237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.361268 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.361260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.361440 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.361294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.361440 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.361322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523f34e0-14f8-493b-ad56-20382baa47be-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.361655 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.361626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.361655 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.361649    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.361755 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.361685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.363324 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.363306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.363713 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.363691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523f34e0-14f8-493b-ad56-20382baa47be-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.370211 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.370189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5df99\" (UniqueName: \"kubernetes.io/projected/523f34e0-14f8-493b-ad56-20382baa47be-kube-api-access-5df99\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.491817 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.491766    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"]
Sep 17 17:01:04.495208 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.495188    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"
Sep 17 17:01:04.500689 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.500666    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-xsr56\""
Sep 17 17:01:04.507199 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.507179    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"]
Sep 17 17:01:04.531911 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.531890    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:04.563164 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.563139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a072fd-2f02-4e19-ba8c-c5e2d30c667b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn\" (UID: \"48a072fd-2f02-4e19-ba8c-c5e2d30c667b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"
Sep 17 17:01:04.563279 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.563222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqxmd\" (UniqueName: \"kubernetes.io/projected/48a072fd-2f02-4e19-ba8c-c5e2d30c667b-kube-api-access-nqxmd\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn\" (UID: \"48a072fd-2f02-4e19-ba8c-c5e2d30c667b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"
Sep 17 17:01:04.664384 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.664357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxmd\" (UniqueName: \"kubernetes.io/projected/48a072fd-2f02-4e19-ba8c-c5e2d30c667b-kube-api-access-nqxmd\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn\" (UID: \"48a072fd-2f02-4e19-ba8c-c5e2d30c667b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"
Sep 17 17:01:04.664533 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.664397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a072fd-2f02-4e19-ba8c-c5e2d30c667b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn\" (UID: \"48a072fd-2f02-4e19-ba8c-c5e2d30c667b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"
Sep 17 17:01:04.666579 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.666553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a072fd-2f02-4e19-ba8c-c5e2d30c667b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn\" (UID: \"48a072fd-2f02-4e19-ba8c-c5e2d30c667b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"
Sep 17 17:01:04.674324 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.674300    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqxmd\" (UniqueName: \"kubernetes.io/projected/48a072fd-2f02-4e19-ba8c-c5e2d30c667b-kube-api-access-nqxmd\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn\" (UID: \"48a072fd-2f02-4e19-ba8c-c5e2d30c667b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"
Sep 17 17:01:04.804898 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.804849    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"
Sep 17 17:01:04.807794 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:04.807772    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"]
Sep 17 17:01:04.810347 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:01:04.810329    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523f34e0_14f8_493b_ad56_20382baa47be.slice/crio-febc82dcc397bc9b56446dd7786c41655baf156c2e7ad1d92768d4d34776096b WatchSource:0}: Error finding container febc82dcc397bc9b56446dd7786c41655baf156c2e7ad1d92768d4d34776096b: Status 404 returned error can't find the container with id febc82dcc397bc9b56446dd7786c41655baf156c2e7ad1d92768d4d34776096b
Sep 17 17:01:05.040862 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:05.040829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2" event={"ID":"523f34e0-14f8-493b-ad56-20382baa47be","Type":"ContainerStarted","Data":"febc82dcc397bc9b56446dd7786c41655baf156c2e7ad1d92768d4d34776096b"}
Sep 17 17:01:05.087990 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:05.087914    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"]
Sep 17 17:01:05.091126 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:01:05.091098    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a072fd_2f02_4e19_ba8c_c5e2d30c667b.slice/crio-a965d8d1466e92601551475f8320d48070b449d3baa307c7864f840311147f58 WatchSource:0}: Error finding container a965d8d1466e92601551475f8320d48070b449d3baa307c7864f840311147f58: Status 404 returned error can't find the container with id a965d8d1466e92601551475f8320d48070b449d3baa307c7864f840311147f58
Sep 17 17:01:06.047481 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:06.047439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn" event={"ID":"48a072fd-2f02-4e19-ba8c-c5e2d30c667b","Type":"ContainerStarted","Data":"3255ba5789815441485a3d043eec3d951986406e38944a01ce2964a7b94d657b"}
Sep 17 17:01:06.047927 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:06.047544    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"
Sep 17 17:01:06.047927 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:06.047576    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn" event={"ID":"48a072fd-2f02-4e19-ba8c-c5e2d30c667b","Type":"ContainerStarted","Data":"a965d8d1466e92601551475f8320d48070b449d3baa307c7864f840311147f58"}
Sep 17 17:01:06.071520 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:06.071474    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn" podStartSLOduration=2.071457761 podStartE2EDuration="2.071457761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:01:06.068826903 +0000 UTC m=+1145.491764767" watchObservedRunningTime="2026-09-17 17:01:06.071457761 +0000 UTC m=+1145.494395624"
Sep 17 17:01:07.053155 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:07.053124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2" event={"ID":"523f34e0-14f8-493b-ad56-20382baa47be","Type":"ContainerStarted","Data":"3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213"}
Sep 17 17:01:07.053613 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:07.053397    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:08.060123 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:08.060086    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2" event={"ID":"523f34e0-14f8-493b-ad56-20382baa47be","Type":"ContainerStarted","Data":"83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f"}
Sep 17 17:01:08.085454 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:08.085401    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2" podStartSLOduration=2.60071107 podStartE2EDuration="4.085383977s" totalImagesPullingTime="1.484672907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:01:08.08348508 +0000 UTC m=+1147.506422960" watchObservedRunningTime="2026-09-17 17:01:08.085383977 +0000 UTC m=+1147.508321840"
Sep 17 17:01:08.532225 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:08.532185    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:14.532392 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:14.532286    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:14.544688 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:14.544655    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:15.104699 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:15.104670    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:16.110890 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:16.110862    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:01:37.055025 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:01:37.054994    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"
Sep 17 17:02:02.793324 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:02.793293    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:02:02.795616 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:02.795595    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:02:08.378040 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:08.378010    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"]
Sep 17 17:02:08.387054 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:08.387025    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"]
Sep 17 17:02:08.395684 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:02:08.395645    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:02:08.395684 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:02:08.395684 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:02:08.395684 ip-10-0-140-49 kubenswrapper[2581]:  > containerID="83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f" cmd=["/bin/sleep","15"]
Sep 17 17:02:08.395906 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:08.395702    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2" message="" err=<
Sep 17 17:02:08.395906 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:02:08.395906 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:02:08.395906 ip-10-0-140-49 kubenswrapper[2581]:  >
Sep 17 17:02:08.395906 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:02:08.395717    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:02:08.395906 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:02:08.395906 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:02:08.395906 ip-10-0-140-49 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2" podUID="523f34e0-14f8-493b-ad56-20382baa47be" containerName="main" containerID="cri-o://83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f"
Sep 17 17:02:08.395906 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:08.395759    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2" podUID="523f34e0-14f8-493b-ad56-20382baa47be" containerName="main" containerID="cri-o://83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f" gracePeriod=60
Sep 17 17:02:08.612475 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:08.612434    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2" podUID="523f34e0-14f8-493b-ad56-20382baa47be" containerName="llm-d-routing-sidecar" containerID="cri-o://3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213" gracePeriod=60
Sep 17 17:02:08.864034 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:08.864009    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:02:09.042306 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.042229    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-model-cache\") pod \"523f34e0-14f8-493b-ad56-20382baa47be\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") "
Sep 17 17:02:09.042306 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.042279    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5df99\" (UniqueName: \"kubernetes.io/projected/523f34e0-14f8-493b-ad56-20382baa47be-kube-api-access-5df99\") pod \"523f34e0-14f8-493b-ad56-20382baa47be\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") "
Sep 17 17:02:09.042519 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.042321    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-dshm\") pod \"523f34e0-14f8-493b-ad56-20382baa47be\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") "
Sep 17 17:02:09.042519 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.042379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523f34e0-14f8-493b-ad56-20382baa47be-tls-certs\") pod \"523f34e0-14f8-493b-ad56-20382baa47be\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") "
Sep 17 17:02:09.042519 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.042403    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-home\") pod \"523f34e0-14f8-493b-ad56-20382baa47be\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") "
Sep 17 17:02:09.042519 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.042436    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-tmp-dir\") pod \"523f34e0-14f8-493b-ad56-20382baa47be\" (UID: \"523f34e0-14f8-493b-ad56-20382baa47be\") "
Sep 17 17:02:09.042519 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.042499    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-model-cache" (OuterVolumeSpecName: "model-cache") pod "523f34e0-14f8-493b-ad56-20382baa47be" (UID: "523f34e0-14f8-493b-ad56-20382baa47be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:09.042771 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.042704    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:02:09.042771 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.042717    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "523f34e0-14f8-493b-ad56-20382baa47be" (UID: "523f34e0-14f8-493b-ad56-20382baa47be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:09.042771 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.042712    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-home" (OuterVolumeSpecName: "home") pod "523f34e0-14f8-493b-ad56-20382baa47be" (UID: "523f34e0-14f8-493b-ad56-20382baa47be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:09.044373 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.044352    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-dshm" (OuterVolumeSpecName: "dshm") pod "523f34e0-14f8-493b-ad56-20382baa47be" (UID: "523f34e0-14f8-493b-ad56-20382baa47be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:09.044462 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.044391    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523f34e0-14f8-493b-ad56-20382baa47be-kube-api-access-5df99" (OuterVolumeSpecName: "kube-api-access-5df99") pod "523f34e0-14f8-493b-ad56-20382baa47be" (UID: "523f34e0-14f8-493b-ad56-20382baa47be"). InnerVolumeSpecName "kube-api-access-5df99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:02:09.044462 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.044451    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/523f34e0-14f8-493b-ad56-20382baa47be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "523f34e0-14f8-493b-ad56-20382baa47be" (UID: "523f34e0-14f8-493b-ad56-20382baa47be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:02:09.143902 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.143868    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:02:09.143902 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.143900    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5df99\" (UniqueName: \"kubernetes.io/projected/523f34e0-14f8-493b-ad56-20382baa47be-kube-api-access-5df99\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:02:09.143902 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.143911    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:02:09.144093 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.143920    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523f34e0-14f8-493b-ad56-20382baa47be-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:02:09.144093 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.143928    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/523f34e0-14f8-493b-ad56-20382baa47be-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:02:09.352210 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.352145    2581 generic.go:356] "Generic (PLEG): container finished" podID="523f34e0-14f8-493b-ad56-20382baa47be" containerID="83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f" exitCode=0
Sep 17 17:02:09.352210 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.352167    2581 generic.go:356] "Generic (PLEG): container finished" podID="523f34e0-14f8-493b-ad56-20382baa47be" containerID="3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213" exitCode=0
Sep 17 17:02:09.352210 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.352201    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"
Sep 17 17:02:09.352398 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.352203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2" event={"ID":"523f34e0-14f8-493b-ad56-20382baa47be","Type":"ContainerDied","Data":"83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f"}
Sep 17 17:02:09.352398 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.352346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2" event={"ID":"523f34e0-14f8-493b-ad56-20382baa47be","Type":"ContainerDied","Data":"3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213"}
Sep 17 17:02:09.352398 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.352382    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2" event={"ID":"523f34e0-14f8-493b-ad56-20382baa47be","Type":"ContainerDied","Data":"febc82dcc397bc9b56446dd7786c41655baf156c2e7ad1d92768d4d34776096b"}
Sep 17 17:02:09.352521 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.352412    2581 scope.go:117] "RemoveContainer" containerID="83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f"
Sep 17 17:02:09.361135 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.361107    2581 scope.go:117] "RemoveContainer" containerID="3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213"
Sep 17 17:02:09.368668 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.368653    2581 scope.go:117] "RemoveContainer" containerID="83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f"
Sep 17 17:02:09.368944 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:02:09.368923    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f\": container with ID starting with 83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f not found: ID does not exist" containerID="83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f"
Sep 17 17:02:09.369024 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.368950    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f"} err="failed to get container status \"83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f\": rpc error: code = NotFound desc = could not find container \"83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f\": container with ID starting with 83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f not found: ID does not exist"
Sep 17 17:02:09.369024 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.369017    2581 scope.go:117] "RemoveContainer" containerID="3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213"
Sep 17 17:02:09.369297 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:02:09.369279    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213\": container with ID starting with 3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213 not found: ID does not exist" containerID="3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213"
Sep 17 17:02:09.369356 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.369306    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213"} err="failed to get container status \"3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213\": rpc error: code = NotFound desc = could not find container \"3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213\": container with ID starting with 3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213 not found: ID does not exist"
Sep 17 17:02:09.369356 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.369331    2581 scope.go:117] "RemoveContainer" containerID="83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f"
Sep 17 17:02:09.369564 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.369546    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f"} err="failed to get container status \"83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f\": rpc error: code = NotFound desc = could not find container \"83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f\": container with ID starting with 83fbe214390b1e7c269856dbec7d3b41d8e795d4dc8487586891aa4be2f2ea4f not found: ID does not exist"
Sep 17 17:02:09.369614 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.369564    2581 scope.go:117] "RemoveContainer" containerID="3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213"
Sep 17 17:02:09.369796 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.369757    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213"} err="failed to get container status \"3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213\": rpc error: code = NotFound desc = could not find container \"3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213\": container with ID starting with 3d5d270f5357be9a2a87ec879848dbba6987aa92f1b9343a17dba2f8446eb213 not found: ID does not exist"
Sep 17 17:02:09.409777 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.409706    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"]
Sep 17 17:02:09.411266 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:09.411248    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-757g2"]
Sep 17 17:02:10.835246 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:10.835204    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="523f34e0-14f8-493b-ad56-20382baa47be" path="/var/lib/kubelet/pods/523f34e0-14f8-493b-ad56-20382baa47be/volumes"
Sep 17 17:02:17.054729 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:17.054678    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn" podUID="48a072fd-2f02-4e19-ba8c-c5e2d30c667b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:02:23.397294 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:23.397237    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn" podUID="48a072fd-2f02-4e19-ba8c-c5e2d30c667b" containerName="main" containerID="cri-o://3255ba5789815441485a3d043eec3d951986406e38944a01ce2964a7b94d657b" gracePeriod=45
Sep 17 17:02:27.055212 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:27.055164    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn" podUID="48a072fd-2f02-4e19-ba8c-c5e2d30c667b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:02:37.054692 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:37.054647    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn" podUID="48a072fd-2f02-4e19-ba8c-c5e2d30c667b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:02:37.055264 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:37.054755    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"
Sep 17 17:02:47.055550 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:47.055505    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn" podUID="48a072fd-2f02-4e19-ba8c-c5e2d30c667b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:02:53.569461 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:53.569426    2581 generic.go:356] "Generic (PLEG): container finished" podID="48a072fd-2f02-4e19-ba8c-c5e2d30c667b" containerID="3255ba5789815441485a3d043eec3d951986406e38944a01ce2964a7b94d657b" exitCode=0
Sep 17 17:02:53.569880 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:53.569492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn" event={"ID":"48a072fd-2f02-4e19-ba8c-c5e2d30c667b","Type":"ContainerDied","Data":"3255ba5789815441485a3d043eec3d951986406e38944a01ce2964a7b94d657b"}
Sep 17 17:02:53.645484 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:53.645465    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"
Sep 17 17:02:53.778567 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:53.778541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a072fd-2f02-4e19-ba8c-c5e2d30c667b-tls-certs\") pod \"48a072fd-2f02-4e19-ba8c-c5e2d30c667b\" (UID: \"48a072fd-2f02-4e19-ba8c-c5e2d30c667b\") "
Sep 17 17:02:53.778694 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:53.778605    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqxmd\" (UniqueName: \"kubernetes.io/projected/48a072fd-2f02-4e19-ba8c-c5e2d30c667b-kube-api-access-nqxmd\") pod \"48a072fd-2f02-4e19-ba8c-c5e2d30c667b\" (UID: \"48a072fd-2f02-4e19-ba8c-c5e2d30c667b\") "
Sep 17 17:02:53.780595 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:53.780565    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a072fd-2f02-4e19-ba8c-c5e2d30c667b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48a072fd-2f02-4e19-ba8c-c5e2d30c667b" (UID: "48a072fd-2f02-4e19-ba8c-c5e2d30c667b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:02:53.780721 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:53.780696    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a072fd-2f02-4e19-ba8c-c5e2d30c667b-kube-api-access-nqxmd" (OuterVolumeSpecName: "kube-api-access-nqxmd") pod "48a072fd-2f02-4e19-ba8c-c5e2d30c667b" (UID: "48a072fd-2f02-4e19-ba8c-c5e2d30c667b"). InnerVolumeSpecName "kube-api-access-nqxmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:02:53.879550 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:53.879524    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqxmd\" (UniqueName: \"kubernetes.io/projected/48a072fd-2f02-4e19-ba8c-c5e2d30c667b-kube-api-access-nqxmd\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:02:53.879550 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:53.879547    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a072fd-2f02-4e19-ba8c-c5e2d30c667b-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:02:54.577653 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:54.577625    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"
Sep 17 17:02:54.578220 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:54.577665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn" event={"ID":"48a072fd-2f02-4e19-ba8c-c5e2d30c667b","Type":"ContainerDied","Data":"a965d8d1466e92601551475f8320d48070b449d3baa307c7864f840311147f58"}
Sep 17 17:02:54.578220 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:54.577748    2581 scope.go:117] "RemoveContainer" containerID="3255ba5789815441485a3d043eec3d951986406e38944a01ce2964a7b94d657b"
Sep 17 17:02:54.601758 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:54.601727    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"]
Sep 17 17:02:54.610038 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:54.610014    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bf9fn"]
Sep 17 17:02:54.835554 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:02:54.835495    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48a072fd-2f02-4e19-ba8c-c5e2d30c667b" path="/var/lib/kubelet/pods/48a072fd-2f02-4e19-ba8c-c5e2d30c667b/volumes"
Sep 17 17:03:05.671789 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.671751    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"]
Sep 17 17:03:05.672246 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.672228    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a072fd-2f02-4e19-ba8c-c5e2d30c667b" containerName="main"
Sep 17 17:03:05.672246 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.672248    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a072fd-2f02-4e19-ba8c-c5e2d30c667b" containerName="main"
Sep 17 17:03:05.672321 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.672262    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="523f34e0-14f8-493b-ad56-20382baa47be" containerName="llm-d-routing-sidecar"
Sep 17 17:03:05.672321 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.672268    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="523f34e0-14f8-493b-ad56-20382baa47be" containerName="llm-d-routing-sidecar"
Sep 17 17:03:05.672321 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.672277    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="523f34e0-14f8-493b-ad56-20382baa47be" containerName="main"
Sep 17 17:03:05.672321 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.672283    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="523f34e0-14f8-493b-ad56-20382baa47be" containerName="main"
Sep 17 17:03:05.672444 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.672339    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="523f34e0-14f8-493b-ad56-20382baa47be" containerName="llm-d-routing-sidecar"
Sep 17 17:03:05.672444 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.672347    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="523f34e0-14f8-493b-ad56-20382baa47be" containerName="main"
Sep 17 17:03:05.672444 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.672356    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a072fd-2f02-4e19-ba8c-c5e2d30c667b" containerName="main"
Sep 17 17:03:05.676893 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.676848    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.681143 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.681121    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 17:03:05.681394 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.681378    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 17:03:05.681608 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.681590    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-xm4gn\""
Sep 17 17:03:05.682326 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.682309    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 17:03:05.685470 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.685451    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-4d4vv\""
Sep 17 17:03:05.700110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.700087    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"]
Sep 17 17:03:05.763629 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.763598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.763725 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.763644    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8e05cc8-3db2-42c9-816e-9be1033bb457-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.763725 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.763665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.763812 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.763759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk2bb\" (UniqueName: \"kubernetes.io/projected/e8e05cc8-3db2-42c9-816e-9be1033bb457-kube-api-access-mk2bb\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.763812 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.763787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.763900 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.763809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.864349 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.864323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk2bb\" (UniqueName: \"kubernetes.io/projected/e8e05cc8-3db2-42c9-816e-9be1033bb457-kube-api-access-mk2bb\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.864440 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.864354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.864440 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.864377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.864440 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.864422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.864572 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.864463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8e05cc8-3db2-42c9-816e-9be1033bb457-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.864572 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.864487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.864811 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.864784    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.864811 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.864801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.864935 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.864852    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.866608 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.866579    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.867021 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.866997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8e05cc8-3db2-42c9-816e-9be1033bb457-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.879381 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.879358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk2bb\" (UniqueName: \"kubernetes.io/projected/e8e05cc8-3db2-42c9-816e-9be1033bb457-kube-api-access-mk2bb\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:05.986672 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:05.986646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:06.091053 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.091027    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"]
Sep 17 17:03:06.094513 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.094497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"
Sep 17 17:03:06.098759 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.098740    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-phtks\""
Sep 17 17:03:06.126570 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.126548    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"]
Sep 17 17:03:06.167432 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.167410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkshx\" (UniqueName: \"kubernetes.io/projected/206a0ede-9acb-43d8-abd8-baf08347f995-kube-api-access-pkshx\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched58f2l\" (UID: \"206a0ede-9acb-43d8-abd8-baf08347f995\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"
Sep 17 17:03:06.167548 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.167456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206a0ede-9acb-43d8-abd8-baf08347f995-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched58f2l\" (UID: \"206a0ede-9acb-43d8-abd8-baf08347f995\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"
Sep 17 17:03:06.268513 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.268457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkshx\" (UniqueName: \"kubernetes.io/projected/206a0ede-9acb-43d8-abd8-baf08347f995-kube-api-access-pkshx\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched58f2l\" (UID: \"206a0ede-9acb-43d8-abd8-baf08347f995\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"
Sep 17 17:03:06.268513 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.268493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206a0ede-9acb-43d8-abd8-baf08347f995-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched58f2l\" (UID: \"206a0ede-9acb-43d8-abd8-baf08347f995\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"
Sep 17 17:03:06.270790 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.270772    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206a0ede-9acb-43d8-abd8-baf08347f995-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched58f2l\" (UID: \"206a0ede-9acb-43d8-abd8-baf08347f995\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"
Sep 17 17:03:06.276642 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.276624    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"]
Sep 17 17:03:06.278944 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:03:06.278919    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e05cc8_3db2_42c9_816e_9be1033bb457.slice/crio-3389b88b0b19775b4fc788fa92001b6311976d692ebb26ace2a0ca6185847d09 WatchSource:0}: Error finding container 3389b88b0b19775b4fc788fa92001b6311976d692ebb26ace2a0ca6185847d09: Status 404 returned error can't find the container with id 3389b88b0b19775b4fc788fa92001b6311976d692ebb26ace2a0ca6185847d09
Sep 17 17:03:06.283220 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.283200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkshx\" (UniqueName: \"kubernetes.io/projected/206a0ede-9acb-43d8-abd8-baf08347f995-kube-api-access-pkshx\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched58f2l\" (UID: \"206a0ede-9acb-43d8-abd8-baf08347f995\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"
Sep 17 17:03:06.405573 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.405551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"
Sep 17 17:03:06.637226 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.637157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk" event={"ID":"e8e05cc8-3db2-42c9-816e-9be1033bb457","Type":"ContainerStarted","Data":"9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21"}
Sep 17 17:03:06.637348 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.637244    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:06.637348 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.637264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk" event={"ID":"e8e05cc8-3db2-42c9-816e-9be1033bb457","Type":"ContainerStarted","Data":"3389b88b0b19775b4fc788fa92001b6311976d692ebb26ace2a0ca6185847d09"}
Sep 17 17:03:06.748175 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:06.748150    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"]
Sep 17 17:03:06.750716 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:03:06.750683    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod206a0ede_9acb_43d8_abd8_baf08347f995.slice/crio-659c2eed7fd35a58f9dc28fa00479824ebac662da5999582aa1d6736590d65b9 WatchSource:0}: Error finding container 659c2eed7fd35a58f9dc28fa00479824ebac662da5999582aa1d6736590d65b9: Status 404 returned error can't find the container with id 659c2eed7fd35a58f9dc28fa00479824ebac662da5999582aa1d6736590d65b9
Sep 17 17:03:07.644575 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:07.644539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk" event={"ID":"e8e05cc8-3db2-42c9-816e-9be1033bb457","Type":"ContainerStarted","Data":"a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586"}
Sep 17 17:03:07.646118 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:07.646088    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l" event={"ID":"206a0ede-9acb-43d8-abd8-baf08347f995","Type":"ContainerStarted","Data":"2d45cf0a98b2ca8e9c70f14a686455bb5d74c127cb9aed164984eea01c98bd2e"}
Sep 17 17:03:07.646242 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:07.646137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l" event={"ID":"206a0ede-9acb-43d8-abd8-baf08347f995","Type":"ContainerStarted","Data":"659c2eed7fd35a58f9dc28fa00479824ebac662da5999582aa1d6736590d65b9"}
Sep 17 17:03:07.646242 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:07.646172    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"
Sep 17 17:03:07.681543 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:07.681500    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk" podStartSLOduration=2.681486598 podStartE2EDuration="2.681486598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:03:07.678837466 +0000 UTC m=+1267.101775328" watchObservedRunningTime="2026-09-17 17:03:07.681486598 +0000 UTC m=+1267.104424484"
Sep 17 17:03:07.717692 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:07.717650    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l" podStartSLOduration=1.717639266 podStartE2EDuration="1.717639266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:03:07.716291819 +0000 UTC m=+1267.139229682" watchObservedRunningTime="2026-09-17 17:03:07.717639266 +0000 UTC m=+1267.140577129"
Sep 17 17:03:07.987496 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:07.987472    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:15.987861 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:15.987821    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:16.000193 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:16.000158    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:16.704330 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:16.704300    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:16.705924 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:16.705895    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:38.652986 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:38.652931    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"
Sep 17 17:03:40.655164 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:40.655132    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"]
Sep 17 17:03:40.658627 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:40.658588    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"]
Sep 17 17:03:40.665942 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:03:40.665889    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:03:40.665942 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:03:40.665942 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:03:40.665942 ip-10-0-140-49 kubenswrapper[2581]:  > containerID="a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586" cmd=["/bin/sleep","15"]
Sep 17 17:03:40.666186 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:40.665984    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk" message="" err=<
Sep 17 17:03:40.666186 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:03:40.666186 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:03:40.666186 ip-10-0-140-49 kubenswrapper[2581]:  >
Sep 17 17:03:40.666186 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:03:40.666006    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:03:40.666186 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:03:40.666186 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:03:40.666186 ip-10-0-140-49 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk" podUID="e8e05cc8-3db2-42c9-816e-9be1033bb457" containerName="main" containerID="cri-o://a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586"
Sep 17 17:03:40.666186 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:40.666057    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk" podUID="e8e05cc8-3db2-42c9-816e-9be1033bb457" containerName="main" containerID="cri-o://a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586" gracePeriod=60
Sep 17 17:03:40.693267 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:40.693238    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk" podUID="e8e05cc8-3db2-42c9-816e-9be1033bb457" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8001/health\": dial tcp 10.134.0.36:8001: connect: connection refused"
Sep 17 17:03:40.881928 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:40.881899    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk" podUID="e8e05cc8-3db2-42c9-816e-9be1033bb457" containerName="llm-d-routing-sidecar" containerID="cri-o://9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21" gracePeriod=60
Sep 17 17:03:41.128031 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.128010    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:41.222864 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.222833    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8e05cc8-3db2-42c9-816e-9be1033bb457-tls-certs\") pod \"e8e05cc8-3db2-42c9-816e-9be1033bb457\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") "
Sep 17 17:03:41.222864 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.222867    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk2bb\" (UniqueName: \"kubernetes.io/projected/e8e05cc8-3db2-42c9-816e-9be1033bb457-kube-api-access-mk2bb\") pod \"e8e05cc8-3db2-42c9-816e-9be1033bb457\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") "
Sep 17 17:03:41.223060 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.222901    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-tmp-dir\") pod \"e8e05cc8-3db2-42c9-816e-9be1033bb457\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") "
Sep 17 17:03:41.223060 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.222931    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-model-cache\") pod \"e8e05cc8-3db2-42c9-816e-9be1033bb457\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") "
Sep 17 17:03:41.223060 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.223019    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-home\") pod \"e8e05cc8-3db2-42c9-816e-9be1033bb457\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") "
Sep 17 17:03:41.223183 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.223064    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-dshm\") pod \"e8e05cc8-3db2-42c9-816e-9be1033bb457\" (UID: \"e8e05cc8-3db2-42c9-816e-9be1033bb457\") "
Sep 17 17:03:41.223242 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.223216    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8e05cc8-3db2-42c9-816e-9be1033bb457" (UID: "e8e05cc8-3db2-42c9-816e-9be1033bb457"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:03:41.223299 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.223250    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-home" (OuterVolumeSpecName: "home") pod "e8e05cc8-3db2-42c9-816e-9be1033bb457" (UID: "e8e05cc8-3db2-42c9-816e-9be1033bb457"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:03:41.223355 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.223287    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8e05cc8-3db2-42c9-816e-9be1033bb457" (UID: "e8e05cc8-3db2-42c9-816e-9be1033bb457"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:03:41.223355 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.223324    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.223355 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.223338    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.225017 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.224996    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-dshm" (OuterVolumeSpecName: "dshm") pod "e8e05cc8-3db2-42c9-816e-9be1033bb457" (UID: "e8e05cc8-3db2-42c9-816e-9be1033bb457"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:03:41.225099 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.225068    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e05cc8-3db2-42c9-816e-9be1033bb457-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8e05cc8-3db2-42c9-816e-9be1033bb457" (UID: "e8e05cc8-3db2-42c9-816e-9be1033bb457"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:03:41.225099 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.225070    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e05cc8-3db2-42c9-816e-9be1033bb457-kube-api-access-mk2bb" (OuterVolumeSpecName: "kube-api-access-mk2bb") pod "e8e05cc8-3db2-42c9-816e-9be1033bb457" (UID: "e8e05cc8-3db2-42c9-816e-9be1033bb457"). InnerVolumeSpecName "kube-api-access-mk2bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:03:41.324158 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.324105    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8e05cc8-3db2-42c9-816e-9be1033bb457-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.324158 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.324127    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk2bb\" (UniqueName: \"kubernetes.io/projected/e8e05cc8-3db2-42c9-816e-9be1033bb457-kube-api-access-mk2bb\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.324158 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.324136    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.324158 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.324144    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8e05cc8-3db2-42c9-816e-9be1033bb457-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.817988 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.817942    2581 generic.go:356] "Generic (PLEG): container finished" podID="e8e05cc8-3db2-42c9-816e-9be1033bb457" containerID="a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586" exitCode=0
Sep 17 17:03:41.817988 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.817988    2581 generic.go:356] "Generic (PLEG): container finished" podID="e8e05cc8-3db2-42c9-816e-9be1033bb457" containerID="9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21" exitCode=0
Sep 17 17:03:41.818390 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.818006    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"
Sep 17 17:03:41.818390 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.818023    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk" event={"ID":"e8e05cc8-3db2-42c9-816e-9be1033bb457","Type":"ContainerDied","Data":"a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586"}
Sep 17 17:03:41.818390 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.818083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk" event={"ID":"e8e05cc8-3db2-42c9-816e-9be1033bb457","Type":"ContainerDied","Data":"9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21"}
Sep 17 17:03:41.818390 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.818111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk" event={"ID":"e8e05cc8-3db2-42c9-816e-9be1033bb457","Type":"ContainerDied","Data":"3389b88b0b19775b4fc788fa92001b6311976d692ebb26ace2a0ca6185847d09"}
Sep 17 17:03:41.818390 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.818145    2581 scope.go:117] "RemoveContainer" containerID="a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586"
Sep 17 17:03:41.828653 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.828631    2581 scope.go:117] "RemoveContainer" containerID="9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21"
Sep 17 17:03:41.836312 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.836296    2581 scope.go:117] "RemoveContainer" containerID="a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586"
Sep 17 17:03:41.836583 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:03:41.836559    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586\": container with ID starting with a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586 not found: ID does not exist" containerID="a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586"
Sep 17 17:03:41.836647 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.836594    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586"} err="failed to get container status \"a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586\": rpc error: code = NotFound desc = could not find container \"a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586\": container with ID starting with a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586 not found: ID does not exist"
Sep 17 17:03:41.836647 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.836617    2581 scope.go:117] "RemoveContainer" containerID="9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21"
Sep 17 17:03:41.836870 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:03:41.836854    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21\": container with ID starting with 9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21 not found: ID does not exist" containerID="9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21"
Sep 17 17:03:41.836919 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.836874    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21"} err="failed to get container status \"9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21\": rpc error: code = NotFound desc = could not find container \"9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21\": container with ID starting with 9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21 not found: ID does not exist"
Sep 17 17:03:41.836919 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.836889    2581 scope.go:117] "RemoveContainer" containerID="a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586"
Sep 17 17:03:41.837121 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.837102    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586"} err="failed to get container status \"a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586\": rpc error: code = NotFound desc = could not find container \"a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586\": container with ID starting with a033a344d6d7de2924bae69f51af5bfe7710baf42ddd4018fd6c6eed06fdb586 not found: ID does not exist"
Sep 17 17:03:41.837121 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.837119    2581 scope.go:117] "RemoveContainer" containerID="9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21"
Sep 17 17:03:41.837367 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.837351    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21"} err="failed to get container status \"9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21\": rpc error: code = NotFound desc = could not find container \"9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21\": container with ID starting with 9a905ece148664b4e4172b62abf4496f7ae78de372cb12237a49073354ab9c21 not found: ID does not exist"
Sep 17 17:03:41.845897 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.845877    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"]
Sep 17 17:03:41.855136 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:41.855117    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-vk8bk"]
Sep 17 17:03:42.835574 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:42.835530    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8e05cc8-3db2-42c9-816e-9be1033bb457" path="/var/lib/kubelet/pods/e8e05cc8-3db2-42c9-816e-9be1033bb457/volumes"
Sep 17 17:03:48.652902 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:48.652852    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l" podUID="206a0ede-9acb-43d8-abd8-baf08347f995" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:03:55.678779 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:55.678710    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l" podUID="206a0ede-9acb-43d8-abd8-baf08347f995" containerName="main" containerID="cri-o://2d45cf0a98b2ca8e9c70f14a686455bb5d74c127cb9aed164984eea01c98bd2e" gracePeriod=45
Sep 17 17:03:58.652888 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:03:58.652839    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l" podUID="206a0ede-9acb-43d8-abd8-baf08347f995" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:04:08.653422 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:08.653378    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l" podUID="206a0ede-9acb-43d8-abd8-baf08347f995" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:04:08.653864 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:08.653486    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"
Sep 17 17:04:18.653001 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:18.652879    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l" podUID="206a0ede-9acb-43d8-abd8-baf08347f995" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:04:25.926516 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:25.926494    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"
Sep 17 17:04:26.039108 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.039044    2581 generic.go:356] "Generic (PLEG): container finished" podID="206a0ede-9acb-43d8-abd8-baf08347f995" containerID="2d45cf0a98b2ca8e9c70f14a686455bb5d74c127cb9aed164984eea01c98bd2e" exitCode=0
Sep 17 17:04:26.039108 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.039089    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"
Sep 17 17:04:26.039257 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.039117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l" event={"ID":"206a0ede-9acb-43d8-abd8-baf08347f995","Type":"ContainerDied","Data":"2d45cf0a98b2ca8e9c70f14a686455bb5d74c127cb9aed164984eea01c98bd2e"}
Sep 17 17:04:26.039257 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.039184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l" event={"ID":"206a0ede-9acb-43d8-abd8-baf08347f995","Type":"ContainerDied","Data":"659c2eed7fd35a58f9dc28fa00479824ebac662da5999582aa1d6736590d65b9"}
Sep 17 17:04:26.039257 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.039207    2581 scope.go:117] "RemoveContainer" containerID="2d45cf0a98b2ca8e9c70f14a686455bb5d74c127cb9aed164984eea01c98bd2e"
Sep 17 17:04:26.047196 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.047180    2581 scope.go:117] "RemoveContainer" containerID="2d45cf0a98b2ca8e9c70f14a686455bb5d74c127cb9aed164984eea01c98bd2e"
Sep 17 17:04:26.047458 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:04:26.047439    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d45cf0a98b2ca8e9c70f14a686455bb5d74c127cb9aed164984eea01c98bd2e\": container with ID starting with 2d45cf0a98b2ca8e9c70f14a686455bb5d74c127cb9aed164984eea01c98bd2e not found: ID does not exist" containerID="2d45cf0a98b2ca8e9c70f14a686455bb5d74c127cb9aed164984eea01c98bd2e"
Sep 17 17:04:26.047499 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.047465    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d45cf0a98b2ca8e9c70f14a686455bb5d74c127cb9aed164984eea01c98bd2e"} err="failed to get container status \"2d45cf0a98b2ca8e9c70f14a686455bb5d74c127cb9aed164984eea01c98bd2e\": rpc error: code = NotFound desc = could not find container \"2d45cf0a98b2ca8e9c70f14a686455bb5d74c127cb9aed164984eea01c98bd2e\": container with ID starting with 2d45cf0a98b2ca8e9c70f14a686455bb5d74c127cb9aed164984eea01c98bd2e not found: ID does not exist"
Sep 17 17:04:26.054760 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.054743    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkshx\" (UniqueName: \"kubernetes.io/projected/206a0ede-9acb-43d8-abd8-baf08347f995-kube-api-access-pkshx\") pod \"206a0ede-9acb-43d8-abd8-baf08347f995\" (UID: \"206a0ede-9acb-43d8-abd8-baf08347f995\") "
Sep 17 17:04:26.054822 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.054767    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206a0ede-9acb-43d8-abd8-baf08347f995-tls-certs\") pod \"206a0ede-9acb-43d8-abd8-baf08347f995\" (UID: \"206a0ede-9acb-43d8-abd8-baf08347f995\") "
Sep 17 17:04:26.056725 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.056698    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/206a0ede-9acb-43d8-abd8-baf08347f995-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "206a0ede-9acb-43d8-abd8-baf08347f995" (UID: "206a0ede-9acb-43d8-abd8-baf08347f995"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:04:26.056725 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.056702    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206a0ede-9acb-43d8-abd8-baf08347f995-kube-api-access-pkshx" (OuterVolumeSpecName: "kube-api-access-pkshx") pod "206a0ede-9acb-43d8-abd8-baf08347f995" (UID: "206a0ede-9acb-43d8-abd8-baf08347f995"). InnerVolumeSpecName "kube-api-access-pkshx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:04:26.156094 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.156068    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkshx\" (UniqueName: \"kubernetes.io/projected/206a0ede-9acb-43d8-abd8-baf08347f995-kube-api-access-pkshx\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:04:26.156094 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.156090    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206a0ede-9acb-43d8-abd8-baf08347f995-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:04:26.362927 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.362901    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"]
Sep 17 17:04:26.367958 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.367933    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched58f2l"]
Sep 17 17:04:26.836591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:26.836549    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="206a0ede-9acb-43d8-abd8-baf08347f995" path="/var/lib/kubelet/pods/206a0ede-9acb-43d8-abd8-baf08347f995/volumes"
Sep 17 17:04:36.913599 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.913561    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"]
Sep 17 17:04:36.914111 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.913903    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e05cc8-3db2-42c9-816e-9be1033bb457" containerName="main"
Sep 17 17:04:36.914111 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.913916    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e05cc8-3db2-42c9-816e-9be1033bb457" containerName="main"
Sep 17 17:04:36.914111 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.913928    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="206a0ede-9acb-43d8-abd8-baf08347f995" containerName="main"
Sep 17 17:04:36.914111 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.913933    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="206a0ede-9acb-43d8-abd8-baf08347f995" containerName="main"
Sep 17 17:04:36.914111 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.913950    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e05cc8-3db2-42c9-816e-9be1033bb457" containerName="llm-d-routing-sidecar"
Sep 17 17:04:36.914111 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.913956    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e05cc8-3db2-42c9-816e-9be1033bb457" containerName="llm-d-routing-sidecar"
Sep 17 17:04:36.914111 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.914037    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e05cc8-3db2-42c9-816e-9be1033bb457" containerName="llm-d-routing-sidecar"
Sep 17 17:04:36.914111 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.914047    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e05cc8-3db2-42c9-816e-9be1033bb457" containerName="main"
Sep 17 17:04:36.914111 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.914054    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="206a0ede-9acb-43d8-abd8-baf08347f995" containerName="main"
Sep 17 17:04:36.918733 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.918707    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:36.921364 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.921330    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:04:36.921586 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.921559    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 17:04:36.921672 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.921358    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 17:04:36.921672 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.921394    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 17:04:36.922432 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.922268    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-24bd8\""
Sep 17 17:04:36.922432 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.922395    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 17:04:36.933463 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:36.933439    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"]
Sep 17 17:04:37.040484 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.040448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-model-cache\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.040618 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.040495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjqtj\" (UniqueName: \"kubernetes.io/projected/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-kube-api-access-pjqtj\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.040618 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.040517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.040618 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.040587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.040618 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.040612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.040775 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.040650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-dshm\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.040775 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.040693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-home\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.040775 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.040707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.141893 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.141858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-model-cache\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.142024 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.141909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjqtj\" (UniqueName: \"kubernetes.io/projected/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-kube-api-access-pjqtj\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.142024 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.141938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.142024 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.141999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.142148 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.142110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.142196 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.142165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-dshm\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.142249 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.142231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-home\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.142303 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.142256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.142363 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.142313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-model-cache\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.142420 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.142366    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.142537 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.142517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.142617 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.142591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-home\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.142930 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.142912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.144446 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.144423    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-dshm\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.144535 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.144506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.152841 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.152822    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjqtj\" (UniqueName: \"kubernetes.io/projected/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-kube-api-access-pjqtj\") pod \"tokenizer-clean-path-test-kserve-7d4446fb49-m4btp\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.185002 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.184921    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"]
Sep 17 17:04:37.189428 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.189409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:04:37.201572 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.201552    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"]
Sep 17 17:04:37.231377 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.231355    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:37.344517 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.344471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f033681-7d1c-4ada-8154-030dc374f07f-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:04:37.344517 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.344522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f033681-7d1c-4ada-8154-030dc374f07f-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:04:37.344686 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.344550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f033681-7d1c-4ada-8154-030dc374f07f-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:04:37.344686 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.344661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f033681-7d1c-4ada-8154-030dc374f07f-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:04:37.445017 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.444992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f033681-7d1c-4ada-8154-030dc374f07f-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:04:37.445135 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.445029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f033681-7d1c-4ada-8154-030dc374f07f-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:04:37.445135 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.445052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f033681-7d1c-4ada-8154-030dc374f07f-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:04:37.445135 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.445106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f033681-7d1c-4ada-8154-030dc374f07f-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:04:37.445406 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.445387    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f033681-7d1c-4ada-8154-030dc374f07f-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:04:37.445456 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.445439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f033681-7d1c-4ada-8154-030dc374f07f-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:04:37.445636 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.445618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f033681-7d1c-4ada-8154-030dc374f07f-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:04:37.447530 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.447514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f033681-7d1c-4ada-8154-030dc374f07f-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:04:37.501523 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.501500    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:04:37.508208 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.508189    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"]
Sep 17 17:04:37.511636 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:04:37.511616    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ff1a71_c3f8_4f84_9b6c_9def7b0ad3e1.slice/crio-74e0754380df3414fe19e66118b49a40d4f09f6dce5e46d434728741bc245832 WatchSource:0}: Error finding container 74e0754380df3414fe19e66118b49a40d4f09f6dce5e46d434728741bc245832: Status 404 returned error can't find the container with id 74e0754380df3414fe19e66118b49a40d4f09f6dce5e46d434728741bc245832
Sep 17 17:04:37.513870 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.513854    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:04:37.783760 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:37.783723    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"]
Sep 17 17:04:38.095618 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:38.095534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp" event={"ID":"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1","Type":"ContainerStarted","Data":"74e0754380df3414fe19e66118b49a40d4f09f6dce5e46d434728741bc245832"}
Sep 17 17:04:38.096869 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:38.096839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7" event={"ID":"7f033681-7d1c-4ada-8154-030dc374f07f","Type":"ContainerStarted","Data":"b4c86e07e1bfa8952d998718af45270eafe89fab33dbf0888aecbc59953b4526"}
Sep 17 17:04:42.125045 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:42.124998    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp" event={"ID":"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1","Type":"ContainerStarted","Data":"f6189e8148fd71070db63e6011ec2018a2f5b2651710e5821ee56eb851870cf6"}
Sep 17 17:04:42.126710 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:42.126678    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7" event={"ID":"7f033681-7d1c-4ada-8154-030dc374f07f","Type":"ContainerStarted","Data":"fee8d20c73642df83be3be51adcf17ebf5bd1c5a7529de57bdc60343f6222ca1"}
Sep 17 17:04:45.152713 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:45.152671    2581 generic.go:356] "Generic (PLEG): container finished" podID="b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" containerID="f6189e8148fd71070db63e6011ec2018a2f5b2651710e5821ee56eb851870cf6" exitCode=0
Sep 17 17:04:45.153199 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:45.152749    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp" event={"ID":"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1","Type":"ContainerDied","Data":"f6189e8148fd71070db63e6011ec2018a2f5b2651710e5821ee56eb851870cf6"}
Sep 17 17:04:45.154620 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:45.154582    2581 generic.go:356] "Generic (PLEG): container finished" podID="7f033681-7d1c-4ada-8154-030dc374f07f" containerID="fee8d20c73642df83be3be51adcf17ebf5bd1c5a7529de57bdc60343f6222ca1" exitCode=0
Sep 17 17:04:45.154769 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:45.154682    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7" event={"ID":"7f033681-7d1c-4ada-8154-030dc374f07f","Type":"ContainerDied","Data":"fee8d20c73642df83be3be51adcf17ebf5bd1c5a7529de57bdc60343f6222ca1"}
Sep 17 17:04:46.163735 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:46.163655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp" event={"ID":"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1","Type":"ContainerStarted","Data":"e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab"}
Sep 17 17:04:46.197144 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:46.197088    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp" podStartSLOduration=3.622653325 podStartE2EDuration="10.197068964s" totalImagesPullingTime="4.049774105s" totalInitContainerRuntime="2.524641534s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:04:46.193413139 +0000 UTC m=+1365.616351007" watchObservedRunningTime="2026-09-17 17:04:46.197068964 +0000 UTC m=+1365.620006826"
Sep 17 17:04:46.231577 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:46.231547    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:47.231593 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:47.231561    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:47.249706 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:47.249681    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:04:48.193542 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:04:48.193508    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:05:31.415849 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:05:31.415767    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7" event={"ID":"7f033681-7d1c-4ada-8154-030dc374f07f","Type":"ContainerStarted","Data":"c91f098c837971fa3415e648ba63ebf66595aad5cae6426d5372d5f3ee904731"}
Sep 17 17:05:31.416370 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:05:31.415998    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:05:31.438358 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:05:31.438297    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7" podStartSLOduration=2.229533857 podStartE2EDuration="54.438281082s" totalImagesPullingTime="49.688019726s" totalInitContainerRuntime="2.520727499s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:04:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:05:31.436776435 +0000 UTC m=+1410.859714297" watchObservedRunningTime="2026-09-17 17:05:31.438281082 +0000 UTC m=+1410.861218988"
Sep 17 17:06:02.432252 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:02.432214    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:06:03.483802 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:03.483775    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7_7f033681-7d1c-4ada-8154-030dc374f07f/main/0.log"
Sep 17 17:06:04.409238 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.409207    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"]
Sep 17 17:06:04.421944 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:06:04.421890    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:06:04.421944 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:06:04.421944 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:06:04.421944 ip-10-0-140-49 kubenswrapper[2581]:  > containerID="e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab" cmd=["/bin/sleep","15"]
Sep 17 17:06:04.422336 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.422310    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp" message="" err=<
Sep 17 17:06:04.422336 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:06:04.422336 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:06:04.422336 ip-10-0-140-49 kubenswrapper[2581]:  >
Sep 17 17:06:04.424987 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:06:04.422537    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:06:04.424987 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:06:04.424987 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:06:04.424987 ip-10-0-140-49 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp" podUID="b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" containerName="main" containerID="cri-o://e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab"
Sep 17 17:06:04.424987 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.422607    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp" podUID="b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" containerName="main" containerID="cri-o://e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab" gracePeriod=60
Sep 17 17:06:04.426004 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.425954    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"]
Sep 17 17:06:04.426436 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.426387    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7" podUID="7f033681-7d1c-4ada-8154-030dc374f07f" containerName="main" containerID="cri-o://c91f098c837971fa3415e648ba63ebf66595aad5cae6426d5372d5f3ee904731" gracePeriod=30
Sep 17 17:06:04.777974 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.777947    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:06:04.900311 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.900278    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-tmp-dir\") pod \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") "
Sep 17 17:06:04.900311 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.900311    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-tls-certs\") pod \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") "
Sep 17 17:06:04.900501 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.900326    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-cabundle-cert\") pod \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") "
Sep 17 17:06:04.900501 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.900372    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-kserve-provision-location\") pod \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") "
Sep 17 17:06:04.900501 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.900396    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-dshm\") pod \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") "
Sep 17 17:06:04.900501 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.900449    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjqtj\" (UniqueName: \"kubernetes.io/projected/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-kube-api-access-pjqtj\") pod \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") "
Sep 17 17:06:04.900501 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.900481    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-model-cache\") pod \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") "
Sep 17 17:06:04.900765 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.900512    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-home\") pod \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\" (UID: \"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1\") "
Sep 17 17:06:04.900765 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.900543    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" (UID: "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:06:04.900765 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.900729    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:06:04.900911 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.900823    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" (UID: "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:06:04.900911 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.900864    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" (UID: "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:06:04.901032 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.900922    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-home" (OuterVolumeSpecName: "home") pod "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" (UID: "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:06:04.902624 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.902592    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-dshm" (OuterVolumeSpecName: "dshm") pod "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" (UID: "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:06:04.902845 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.902814    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" (UID: "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:06:04.902952 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.902873    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-kube-api-access-pjqtj" (OuterVolumeSpecName: "kube-api-access-pjqtj") pod "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" (UID: "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1"). InnerVolumeSpecName "kube-api-access-pjqtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:06:04.919779 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:04.919755    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" (UID: "b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:06:05.001707 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.001676    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjqtj\" (UniqueName: \"kubernetes.io/projected/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-kube-api-access-pjqtj\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:06:05.001810 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.001712    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:06:05.001810 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.001725    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:06:05.001810 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.001737    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:06:05.001810 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.001749    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:06:05.001810 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.001762    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:06:05.001810 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.001774    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:06:05.068744 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.068721    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:06:05.103055 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.103026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f033681-7d1c-4ada-8154-030dc374f07f-tls-certs\") pod \"7f033681-7d1c-4ada-8154-030dc374f07f\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") "
Sep 17 17:06:05.103349 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.103106    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f033681-7d1c-4ada-8154-030dc374f07f-tmp-dir\") pod \"7f033681-7d1c-4ada-8154-030dc374f07f\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") "
Sep 17 17:06:05.103349 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.103155    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f033681-7d1c-4ada-8154-030dc374f07f-kserve-provision-location\") pod \"7f033681-7d1c-4ada-8154-030dc374f07f\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") "
Sep 17 17:06:05.103349 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.103184    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f033681-7d1c-4ada-8154-030dc374f07f-cabundle-cert\") pod \"7f033681-7d1c-4ada-8154-030dc374f07f\" (UID: \"7f033681-7d1c-4ada-8154-030dc374f07f\") "
Sep 17 17:06:05.103531 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.103505    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f033681-7d1c-4ada-8154-030dc374f07f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f033681-7d1c-4ada-8154-030dc374f07f" (UID: "7f033681-7d1c-4ada-8154-030dc374f07f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:06:05.103531 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.103514    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f033681-7d1c-4ada-8154-030dc374f07f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f033681-7d1c-4ada-8154-030dc374f07f" (UID: "7f033681-7d1c-4ada-8154-030dc374f07f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:06:05.105047 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.105025    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f033681-7d1c-4ada-8154-030dc374f07f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f033681-7d1c-4ada-8154-030dc374f07f" (UID: "7f033681-7d1c-4ada-8154-030dc374f07f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:06:05.122786 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.122759    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f033681-7d1c-4ada-8154-030dc374f07f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f033681-7d1c-4ada-8154-030dc374f07f" (UID: "7f033681-7d1c-4ada-8154-030dc374f07f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:06:05.204152 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.204128    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f033681-7d1c-4ada-8154-030dc374f07f-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:06:05.204152 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.204148    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f033681-7d1c-4ada-8154-030dc374f07f-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:06:05.204289 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.204158    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f033681-7d1c-4ada-8154-030dc374f07f-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:06:05.204289 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.204166    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f033681-7d1c-4ada-8154-030dc374f07f-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:06:05.581976 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.581932    2581 generic.go:356] "Generic (PLEG): container finished" podID="b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" containerID="e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab" exitCode=0
Sep 17 17:06:05.582137 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.582003    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"
Sep 17 17:06:05.582137 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.582004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp" event={"ID":"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1","Type":"ContainerDied","Data":"e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab"}
Sep 17 17:06:05.582137 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.582070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp" event={"ID":"b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1","Type":"ContainerDied","Data":"74e0754380df3414fe19e66118b49a40d4f09f6dce5e46d434728741bc245832"}
Sep 17 17:06:05.582137 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.582102    2581 scope.go:117] "RemoveContainer" containerID="e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab"
Sep 17 17:06:05.583856 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.583833    2581 generic.go:356] "Generic (PLEG): container finished" podID="7f033681-7d1c-4ada-8154-030dc374f07f" containerID="c91f098c837971fa3415e648ba63ebf66595aad5cae6426d5372d5f3ee904731" exitCode=0
Sep 17 17:06:05.583955 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.583893    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"
Sep 17 17:06:05.583955 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.583919    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7" event={"ID":"7f033681-7d1c-4ada-8154-030dc374f07f","Type":"ContainerDied","Data":"c91f098c837971fa3415e648ba63ebf66595aad5cae6426d5372d5f3ee904731"}
Sep 17 17:06:05.583955 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.583953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7" event={"ID":"7f033681-7d1c-4ada-8154-030dc374f07f","Type":"ContainerDied","Data":"b4c86e07e1bfa8952d998718af45270eafe89fab33dbf0888aecbc59953b4526"}
Sep 17 17:06:05.591048 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.591029    2581 scope.go:117] "RemoveContainer" containerID="f6189e8148fd71070db63e6011ec2018a2f5b2651710e5821ee56eb851870cf6"
Sep 17 17:06:05.598770 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.598755    2581 scope.go:117] "RemoveContainer" containerID="e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab"
Sep 17 17:06:05.599012 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:06:05.598994    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab\": container with ID starting with e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab not found: ID does not exist" containerID="e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab"
Sep 17 17:06:05.599074 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.599018    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab"} err="failed to get container status \"e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab\": rpc error: code = NotFound desc = could not find container \"e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab\": container with ID starting with e8c3ab56c8ef85162761ed90b64abe13ece4690efeb98a12d0c02a2483eb78ab not found: ID does not exist"
Sep 17 17:06:05.599074 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.599034    2581 scope.go:117] "RemoveContainer" containerID="f6189e8148fd71070db63e6011ec2018a2f5b2651710e5821ee56eb851870cf6"
Sep 17 17:06:05.599269 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:06:05.599251    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6189e8148fd71070db63e6011ec2018a2f5b2651710e5821ee56eb851870cf6\": container with ID starting with f6189e8148fd71070db63e6011ec2018a2f5b2651710e5821ee56eb851870cf6 not found: ID does not exist" containerID="f6189e8148fd71070db63e6011ec2018a2f5b2651710e5821ee56eb851870cf6"
Sep 17 17:06:05.599336 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.599280    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6189e8148fd71070db63e6011ec2018a2f5b2651710e5821ee56eb851870cf6"} err="failed to get container status \"f6189e8148fd71070db63e6011ec2018a2f5b2651710e5821ee56eb851870cf6\": rpc error: code = NotFound desc = could not find container \"f6189e8148fd71070db63e6011ec2018a2f5b2651710e5821ee56eb851870cf6\": container with ID starting with f6189e8148fd71070db63e6011ec2018a2f5b2651710e5821ee56eb851870cf6 not found: ID does not exist"
Sep 17 17:06:05.599336 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.599302    2581 scope.go:117] "RemoveContainer" containerID="c91f098c837971fa3415e648ba63ebf66595aad5cae6426d5372d5f3ee904731"
Sep 17 17:06:05.606332 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.606313    2581 scope.go:117] "RemoveContainer" containerID="fee8d20c73642df83be3be51adcf17ebf5bd1c5a7529de57bdc60343f6222ca1"
Sep 17 17:06:05.613437 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.613415    2581 scope.go:117] "RemoveContainer" containerID="c91f098c837971fa3415e648ba63ebf66595aad5cae6426d5372d5f3ee904731"
Sep 17 17:06:05.613689 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:06:05.613672    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91f098c837971fa3415e648ba63ebf66595aad5cae6426d5372d5f3ee904731\": container with ID starting with c91f098c837971fa3415e648ba63ebf66595aad5cae6426d5372d5f3ee904731 not found: ID does not exist" containerID="c91f098c837971fa3415e648ba63ebf66595aad5cae6426d5372d5f3ee904731"
Sep 17 17:06:05.613739 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.613695    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91f098c837971fa3415e648ba63ebf66595aad5cae6426d5372d5f3ee904731"} err="failed to get container status \"c91f098c837971fa3415e648ba63ebf66595aad5cae6426d5372d5f3ee904731\": rpc error: code = NotFound desc = could not find container \"c91f098c837971fa3415e648ba63ebf66595aad5cae6426d5372d5f3ee904731\": container with ID starting with c91f098c837971fa3415e648ba63ebf66595aad5cae6426d5372d5f3ee904731 not found: ID does not exist"
Sep 17 17:06:05.613739 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.613710    2581 scope.go:117] "RemoveContainer" containerID="fee8d20c73642df83be3be51adcf17ebf5bd1c5a7529de57bdc60343f6222ca1"
Sep 17 17:06:05.613943 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:06:05.613926    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee8d20c73642df83be3be51adcf17ebf5bd1c5a7529de57bdc60343f6222ca1\": container with ID starting with fee8d20c73642df83be3be51adcf17ebf5bd1c5a7529de57bdc60343f6222ca1 not found: ID does not exist" containerID="fee8d20c73642df83be3be51adcf17ebf5bd1c5a7529de57bdc60343f6222ca1"
Sep 17 17:06:05.613998 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.613950    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee8d20c73642df83be3be51adcf17ebf5bd1c5a7529de57bdc60343f6222ca1"} err="failed to get container status \"fee8d20c73642df83be3be51adcf17ebf5bd1c5a7529de57bdc60343f6222ca1\": rpc error: code = NotFound desc = could not find container \"fee8d20c73642df83be3be51adcf17ebf5bd1c5a7529de57bdc60343f6222ca1\": container with ID starting with fee8d20c73642df83be3be51adcf17ebf5bd1c5a7529de57bdc60343f6222ca1 not found: ID does not exist"
Sep 17 17:06:05.619147 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.619129    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"]
Sep 17 17:06:05.625635 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.625616    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7d4446fb49-m4btp"]
Sep 17 17:06:05.640808 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.640789    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"]
Sep 17 17:06:05.644253 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:05.644234    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f56767b68-sw9z7"]
Sep 17 17:06:06.835065 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:06.835032    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f033681-7d1c-4ada-8154-030dc374f07f" path="/var/lib/kubelet/pods/7f033681-7d1c-4ada-8154-030dc374f07f/volumes"
Sep 17 17:06:06.835656 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:06.835634    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" path="/var/lib/kubelet/pods/b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1/volumes"
Sep 17 17:06:24.634328 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.634292    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"]
Sep 17 17:06:24.634907 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.634802    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" containerName="storage-initializer"
Sep 17 17:06:24.634907 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.634826    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" containerName="storage-initializer"
Sep 17 17:06:24.634907 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.634843    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" containerName="main"
Sep 17 17:06:24.634907 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.634853    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" containerName="main"
Sep 17 17:06:24.634907 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.634866    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f033681-7d1c-4ada-8154-030dc374f07f" containerName="main"
Sep 17 17:06:24.634907 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.634874    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f033681-7d1c-4ada-8154-030dc374f07f" containerName="main"
Sep 17 17:06:24.634907 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.634901    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f033681-7d1c-4ada-8154-030dc374f07f" containerName="storage-initializer"
Sep 17 17:06:24.634907 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.634910    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f033681-7d1c-4ada-8154-030dc374f07f" containerName="storage-initializer"
Sep 17 17:06:24.635400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.635026    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3ff1a71-c3f8-4f84-9b6c-9def7b0ad3e1" containerName="main"
Sep 17 17:06:24.635400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.635043    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f033681-7d1c-4ada-8154-030dc374f07f" containerName="main"
Sep 17 17:06:24.637182 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.637158    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"
Sep 17 17:06:24.640766 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.640732    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 17:06:24.641052 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.640784    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-q67zm\""
Sep 17 17:06:24.641052 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.640836    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-c522d\""
Sep 17 17:06:24.641052 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.640845    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 17:06:24.641052 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.640874    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 17:06:24.648504 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.648481    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"]
Sep 17 17:06:24.743803 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.743777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5afe50b-814b-4582-a81d-4d98c31d85fc-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965\" (UID: \"a5afe50b-814b-4582-a81d-4d98c31d85fc\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"
Sep 17 17:06:24.743918 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.743858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5hpv\" (UniqueName: \"kubernetes.io/projected/a5afe50b-814b-4582-a81d-4d98c31d85fc-kube-api-access-c5hpv\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965\" (UID: \"a5afe50b-814b-4582-a81d-4d98c31d85fc\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"
Sep 17 17:06:24.844812 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.844791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5hpv\" (UniqueName: \"kubernetes.io/projected/a5afe50b-814b-4582-a81d-4d98c31d85fc-kube-api-access-c5hpv\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965\" (UID: \"a5afe50b-814b-4582-a81d-4d98c31d85fc\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"
Sep 17 17:06:24.844925 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.844831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5afe50b-814b-4582-a81d-4d98c31d85fc-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965\" (UID: \"a5afe50b-814b-4582-a81d-4d98c31d85fc\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"
Sep 17 17:06:24.847344 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.847323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5afe50b-814b-4582-a81d-4d98c31d85fc-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965\" (UID: \"a5afe50b-814b-4582-a81d-4d98c31d85fc\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"
Sep 17 17:06:24.858241 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.858217    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5hpv\" (UniqueName: \"kubernetes.io/projected/a5afe50b-814b-4582-a81d-4d98c31d85fc-kube-api-access-c5hpv\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965\" (UID: \"a5afe50b-814b-4582-a81d-4d98c31d85fc\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"
Sep 17 17:06:24.949822 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:24.949802    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"
Sep 17 17:06:25.222915 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:25.222893    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"]
Sep 17 17:06:25.225087 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:06:25.225058    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5afe50b_814b_4582_a81d_4d98c31d85fc.slice/crio-01fb1527ef25df4062b50e9da10da086419db53ae73206f423c511cadae48527 WatchSource:0}: Error finding container 01fb1527ef25df4062b50e9da10da086419db53ae73206f423c511cadae48527: Status 404 returned error can't find the container with id 01fb1527ef25df4062b50e9da10da086419db53ae73206f423c511cadae48527
Sep 17 17:06:25.684516 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:25.684418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965" event={"ID":"a5afe50b-814b-4582-a81d-4d98c31d85fc","Type":"ContainerStarted","Data":"1d495742d72645e675312b130412fb58233b0eefa5fe5d0f5e7b519c3895cb66"}
Sep 17 17:06:25.684516 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:25.684479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965" event={"ID":"a5afe50b-814b-4582-a81d-4d98c31d85fc","Type":"ContainerStarted","Data":"01fb1527ef25df4062b50e9da10da086419db53ae73206f423c511cadae48527"}
Sep 17 17:06:25.684516 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:25.684506    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"
Sep 17 17:06:25.712027 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:25.711958    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965" podStartSLOduration=1.711944245 podStartE2EDuration="1.711944245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:06:25.709777734 +0000 UTC m=+1465.132715598" watchObservedRunningTime="2026-09-17 17:06:25.711944245 +0000 UTC m=+1465.134882108"
Sep 17 17:06:56.690343 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:06:56.690309    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"
Sep 17 17:07:02.822027 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:02.821987    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:07:02.823402 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:02.823375    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:07:07.316593 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.316560    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"]
Sep 17 17:07:07.319845 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.319823    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.322151 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.322130    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:07:07.322264 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.322133    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 17:07:07.329906 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.329889    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 17:07:07.330029 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.329907    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 17:07:07.330029 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.329916    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 17:07:07.330029 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.329929    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-mmz5d\""
Sep 17 17:07:07.333310 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.333293    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"]
Sep 17 17:07:07.363866 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.363846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20805d1d-8d83-4526-af1a-ca2b056f8bb9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.363996 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.363873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-model-cache\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.363996 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.363900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20805d1d-8d83-4526-af1a-ca2b056f8bb9-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.363996 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.363931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-home\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.364123 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.364004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.364123 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.364027    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcqq6\" (UniqueName: \"kubernetes.io/projected/20805d1d-8d83-4526-af1a-ca2b056f8bb9-kube-api-access-fcqq6\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.364123 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.364053    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.364123 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.364078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-dshm\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.464376 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.464350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqq6\" (UniqueName: \"kubernetes.io/projected/20805d1d-8d83-4526-af1a-ca2b056f8bb9-kube-api-access-fcqq6\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.464505 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.464387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.464505 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.464411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-dshm\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.464505 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.464434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20805d1d-8d83-4526-af1a-ca2b056f8bb9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.464505 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.464456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-model-cache\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.464505 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.464493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20805d1d-8d83-4526-af1a-ca2b056f8bb9-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.464762 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.464527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-home\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.464762 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.464577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.464869 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.464802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.464948 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.464875    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.465040 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.464946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-home\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.465040 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.465029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-model-cache\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.465281 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.465257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20805d1d-8d83-4526-af1a-ca2b056f8bb9-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.466616 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.466593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-dshm\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.467067 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.467046    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20805d1d-8d83-4526-af1a-ca2b056f8bb9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.475058 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.475038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcqq6\" (UniqueName: \"kubernetes.io/projected/20805d1d-8d83-4526-af1a-ca2b056f8bb9-kube-api-access-fcqq6\") pod \"tokenizer-migration-path-test-kserve-6c49c77444-ndz5d\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.595786 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.595711    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"]
Sep 17 17:07:07.598669 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.598650    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:07.611175 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.611151    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"]
Sep 17 17:07:07.629751 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.629728    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:07.667229 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.666376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/018c3162-3cfe-49d4-a666-0a27085f59a9-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:07.667229 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.666522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/018c3162-3cfe-49d4-a666-0a27085f59a9-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:07.667229 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.666554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/018c3162-3cfe-49d4-a666-0a27085f59a9-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:07.667229 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.666615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/018c3162-3cfe-49d4-a666-0a27085f59a9-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:07.767644 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.767609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/018c3162-3cfe-49d4-a666-0a27085f59a9-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:07.767794 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.767670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/018c3162-3cfe-49d4-a666-0a27085f59a9-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:07.767794 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.767698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/018c3162-3cfe-49d4-a666-0a27085f59a9-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:07.767794 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.767717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/018c3162-3cfe-49d4-a666-0a27085f59a9-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:07.768134 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.768115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/018c3162-3cfe-49d4-a666-0a27085f59a9-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:07.768225 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.768206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/018c3162-3cfe-49d4-a666-0a27085f59a9-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:07.768446 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.768428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/018c3162-3cfe-49d4-a666-0a27085f59a9-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:07.770107 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.770090    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/018c3162-3cfe-49d4-a666-0a27085f59a9-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:07.909273 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.909193    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:07.937472 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:07.937439    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"]
Sep 17 17:07:07.941459 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:07:07.941429    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20805d1d_8d83_4526_af1a_ca2b056f8bb9.slice/crio-d0a2674722322050aa01ec02ee7de51c29a9235989e036a896fa1701e4e14fcf WatchSource:0}: Error finding container d0a2674722322050aa01ec02ee7de51c29a9235989e036a896fa1701e4e14fcf: Status 404 returned error can't find the container with id d0a2674722322050aa01ec02ee7de51c29a9235989e036a896fa1701e4e14fcf
Sep 17 17:07:08.201892 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:08.201858    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"]
Sep 17 17:07:08.205713 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:07:08.205683    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018c3162_3cfe_49d4_a666_0a27085f59a9.slice/crio-2b8fd221b074f89ebb4f2e977db7e399c235cac2b3cdb8b3fcdd39770e20218e WatchSource:0}: Error finding container 2b8fd221b074f89ebb4f2e977db7e399c235cac2b3cdb8b3fcdd39770e20218e: Status 404 returned error can't find the container with id 2b8fd221b074f89ebb4f2e977db7e399c235cac2b3cdb8b3fcdd39770e20218e
Sep 17 17:07:08.889782 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:08.889736    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8" event={"ID":"018c3162-3cfe-49d4-a666-0a27085f59a9","Type":"ContainerStarted","Data":"1783aa04dcec8e2e969e9b3337e67e8b514d45863e821f5b9abf7e6b377be297"}
Sep 17 17:07:08.890306 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:08.889809    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8" event={"ID":"018c3162-3cfe-49d4-a666-0a27085f59a9","Type":"ContainerStarted","Data":"2b8fd221b074f89ebb4f2e977db7e399c235cac2b3cdb8b3fcdd39770e20218e"}
Sep 17 17:07:08.891777 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:08.891751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d" event={"ID":"20805d1d-8d83-4526-af1a-ca2b056f8bb9","Type":"ContainerStarted","Data":"70e9cf7ba69f3e98343f411387445dc8e377904702a918bcfd26a23a20eedbd4"}
Sep 17 17:07:08.891895 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:08.891792    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d" event={"ID":"20805d1d-8d83-4526-af1a-ca2b056f8bb9","Type":"ContainerStarted","Data":"d0a2674722322050aa01ec02ee7de51c29a9235989e036a896fa1701e4e14fcf"}
Sep 17 17:07:10.902614 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:10.902531    2581 generic.go:356] "Generic (PLEG): container finished" podID="20805d1d-8d83-4526-af1a-ca2b056f8bb9" containerID="70e9cf7ba69f3e98343f411387445dc8e377904702a918bcfd26a23a20eedbd4" exitCode=0
Sep 17 17:07:10.903114 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:10.902609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d" event={"ID":"20805d1d-8d83-4526-af1a-ca2b056f8bb9","Type":"ContainerDied","Data":"70e9cf7ba69f3e98343f411387445dc8e377904702a918bcfd26a23a20eedbd4"}
Sep 17 17:07:10.904228 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:10.904205    2581 generic.go:356] "Generic (PLEG): container finished" podID="018c3162-3cfe-49d4-a666-0a27085f59a9" containerID="1783aa04dcec8e2e969e9b3337e67e8b514d45863e821f5b9abf7e6b377be297" exitCode=0
Sep 17 17:07:10.904331 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:10.904274    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8" event={"ID":"018c3162-3cfe-49d4-a666-0a27085f59a9","Type":"ContainerDied","Data":"1783aa04dcec8e2e969e9b3337e67e8b514d45863e821f5b9abf7e6b377be297"}
Sep 17 17:07:11.912520 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:11.912483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d" event={"ID":"20805d1d-8d83-4526-af1a-ca2b056f8bb9","Type":"ContainerStarted","Data":"6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8"}
Sep 17 17:07:11.914404 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:11.914380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8" event={"ID":"018c3162-3cfe-49d4-a666-0a27085f59a9","Type":"ContainerStarted","Data":"9b2d4e58f3155bada024cec69089e233d23b9ae375acbc12a861ac3312006b4f"}
Sep 17 17:07:11.914592 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:11.914576    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:11.937333 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:11.937279    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d" podStartSLOduration=2.397928326 podStartE2EDuration="4.937260302s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.539331976s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:07:11.936466021 +0000 UTC m=+1511.359403884" watchObservedRunningTime="2026-09-17 17:07:11.937260302 +0000 UTC m=+1511.360198184"
Sep 17 17:07:11.961531 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:11.961491    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8" podStartSLOduration=2.455698248 podStartE2EDuration="4.961478819s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.505780571s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:07:11.960724326 +0000 UTC m=+1511.383662201" watchObservedRunningTime="2026-09-17 17:07:11.961478819 +0000 UTC m=+1511.384416682"
Sep 17 17:07:12.630338 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:12.630304    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:17.630648 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:17.630545    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:17.643301 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:17.643276    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:17.953620 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:17.953585    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:42.929921 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:42.929880    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:44.116945 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:44.116910    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8_018c3162-3cfe-49d4-a666-0a27085f59a9/main/0.log"
Sep 17 17:07:45.062317 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.062282    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"]
Sep 17 17:07:45.071626 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:07:45.071590    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:07:45.071626 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:07:45.071626 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:07:45.071626 ip-10-0-140-49 kubenswrapper[2581]:  > containerID="6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8" cmd=["/bin/sleep","15"]
Sep 17 17:07:45.071905 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.071644    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d" message="" err=<
Sep 17 17:07:45.071905 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:07:45.071905 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:07:45.071905 ip-10-0-140-49 kubenswrapper[2581]:  >
Sep 17 17:07:45.071905 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:07:45.071662    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:07:45.071905 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:07:45.071905 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:07:45.071905 ip-10-0-140-49 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d" podUID="20805d1d-8d83-4526-af1a-ca2b056f8bb9" containerName="main" containerID="cri-o://6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8"
Sep 17 17:07:45.071905 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.071707    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d" podUID="20805d1d-8d83-4526-af1a-ca2b056f8bb9" containerName="main" containerID="cri-o://6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8" gracePeriod=60
Sep 17 17:07:45.080086 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.080060    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"]
Sep 17 17:07:45.080296 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.080274    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8" podUID="018c3162-3cfe-49d4-a666-0a27085f59a9" containerName="main" containerID="cri-o://9b2d4e58f3155bada024cec69089e233d23b9ae375acbc12a861ac3312006b4f" gracePeriod=30
Sep 17 17:07:45.433193 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.433165    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:45.609556 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.609475    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-tmp-dir\") pod \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") "
Sep 17 17:07:45.609556 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.609530    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-home\") pod \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") "
Sep 17 17:07:45.609556 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.609554    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20805d1d-8d83-4526-af1a-ca2b056f8bb9-tls-certs\") pod \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") "
Sep 17 17:07:45.609862 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.609573    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcqq6\" (UniqueName: \"kubernetes.io/projected/20805d1d-8d83-4526-af1a-ca2b056f8bb9-kube-api-access-fcqq6\") pod \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") "
Sep 17 17:07:45.609862 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.609588    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-kserve-provision-location\") pod \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") "
Sep 17 17:07:45.609862 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.609608    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-dshm\") pod \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") "
Sep 17 17:07:45.609862 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.609635    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-model-cache\") pod \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") "
Sep 17 17:07:45.609862 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.609752    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20805d1d-8d83-4526-af1a-ca2b056f8bb9" (UID: "20805d1d-8d83-4526-af1a-ca2b056f8bb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:07:45.610166 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.609931    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-home" (OuterVolumeSpecName: "home") pod "20805d1d-8d83-4526-af1a-ca2b056f8bb9" (UID: "20805d1d-8d83-4526-af1a-ca2b056f8bb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:07:45.610166 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.610108    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-model-cache" (OuterVolumeSpecName: "model-cache") pod "20805d1d-8d83-4526-af1a-ca2b056f8bb9" (UID: "20805d1d-8d83-4526-af1a-ca2b056f8bb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:07:45.610698 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.610667    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20805d1d-8d83-4526-af1a-ca2b056f8bb9-cabundle-cert\") pod \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\" (UID: \"20805d1d-8d83-4526-af1a-ca2b056f8bb9\") "
Sep 17 17:07:45.611116 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.611085    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20805d1d-8d83-4526-af1a-ca2b056f8bb9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "20805d1d-8d83-4526-af1a-ca2b056f8bb9" (UID: "20805d1d-8d83-4526-af1a-ca2b056f8bb9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:07:45.611217 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.611157    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:07:45.611217 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.611178    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:07:45.611217 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.611191    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:07:45.612415 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.612391    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20805d1d-8d83-4526-af1a-ca2b056f8bb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20805d1d-8d83-4526-af1a-ca2b056f8bb9" (UID: "20805d1d-8d83-4526-af1a-ca2b056f8bb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:07:45.612755 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.612644    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20805d1d-8d83-4526-af1a-ca2b056f8bb9-kube-api-access-fcqq6" (OuterVolumeSpecName: "kube-api-access-fcqq6") pod "20805d1d-8d83-4526-af1a-ca2b056f8bb9" (UID: "20805d1d-8d83-4526-af1a-ca2b056f8bb9"). InnerVolumeSpecName "kube-api-access-fcqq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:07:45.613461 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.613421    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-dshm" (OuterVolumeSpecName: "dshm") pod "20805d1d-8d83-4526-af1a-ca2b056f8bb9" (UID: "20805d1d-8d83-4526-af1a-ca2b056f8bb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:07:45.634020 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.633981    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "20805d1d-8d83-4526-af1a-ca2b056f8bb9" (UID: "20805d1d-8d83-4526-af1a-ca2b056f8bb9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:07:45.711699 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.711670    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20805d1d-8d83-4526-af1a-ca2b056f8bb9-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:07:45.711699 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.711699    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcqq6\" (UniqueName: \"kubernetes.io/projected/20805d1d-8d83-4526-af1a-ca2b056f8bb9-kube-api-access-fcqq6\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:07:45.711872 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.711710    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:07:45.711872 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.711719    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20805d1d-8d83-4526-af1a-ca2b056f8bb9-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:07:45.711872 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.711727    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20805d1d-8d83-4526-af1a-ca2b056f8bb9-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:07:45.718462 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.718441    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:45.812199 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.812172    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/018c3162-3cfe-49d4-a666-0a27085f59a9-tls-certs\") pod \"018c3162-3cfe-49d4-a666-0a27085f59a9\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") "
Sep 17 17:07:45.812323 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.812211    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/018c3162-3cfe-49d4-a666-0a27085f59a9-kserve-provision-location\") pod \"018c3162-3cfe-49d4-a666-0a27085f59a9\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") "
Sep 17 17:07:45.812323 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.812257    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/018c3162-3cfe-49d4-a666-0a27085f59a9-tmp-dir\") pod \"018c3162-3cfe-49d4-a666-0a27085f59a9\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") "
Sep 17 17:07:45.812323 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.812284    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/018c3162-3cfe-49d4-a666-0a27085f59a9-cabundle-cert\") pod \"018c3162-3cfe-49d4-a666-0a27085f59a9\" (UID: \"018c3162-3cfe-49d4-a666-0a27085f59a9\") "
Sep 17 17:07:45.812663 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.812640    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018c3162-3cfe-49d4-a666-0a27085f59a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "018c3162-3cfe-49d4-a666-0a27085f59a9" (UID: "018c3162-3cfe-49d4-a666-0a27085f59a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:07:45.812740 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.812711    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018c3162-3cfe-49d4-a666-0a27085f59a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "018c3162-3cfe-49d4-a666-0a27085f59a9" (UID: "018c3162-3cfe-49d4-a666-0a27085f59a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:07:45.814250 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.814231    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/018c3162-3cfe-49d4-a666-0a27085f59a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "018c3162-3cfe-49d4-a666-0a27085f59a9" (UID: "018c3162-3cfe-49d4-a666-0a27085f59a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:07:45.833389 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.833354    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018c3162-3cfe-49d4-a666-0a27085f59a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "018c3162-3cfe-49d4-a666-0a27085f59a9" (UID: "018c3162-3cfe-49d4-a666-0a27085f59a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:07:45.913714 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.913639    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/018c3162-3cfe-49d4-a666-0a27085f59a9-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:07:45.913714 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.913670    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/018c3162-3cfe-49d4-a666-0a27085f59a9-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:07:45.913714 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.913678    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/018c3162-3cfe-49d4-a666-0a27085f59a9-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:07:45.913714 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:45.913687    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/018c3162-3cfe-49d4-a666-0a27085f59a9-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:07:46.082820 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.082781    2581 generic.go:356] "Generic (PLEG): container finished" podID="20805d1d-8d83-4526-af1a-ca2b056f8bb9" containerID="6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8" exitCode=0
Sep 17 17:07:46.083033 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.082851    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"
Sep 17 17:07:46.083033 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.082862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d" event={"ID":"20805d1d-8d83-4526-af1a-ca2b056f8bb9","Type":"ContainerDied","Data":"6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8"}
Sep 17 17:07:46.083033 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.082948    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d" event={"ID":"20805d1d-8d83-4526-af1a-ca2b056f8bb9","Type":"ContainerDied","Data":"d0a2674722322050aa01ec02ee7de51c29a9235989e036a896fa1701e4e14fcf"}
Sep 17 17:07:46.083033 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.082998    2581 scope.go:117] "RemoveContainer" containerID="6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8"
Sep 17 17:07:46.084534 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.084509    2581 generic.go:356] "Generic (PLEG): container finished" podID="018c3162-3cfe-49d4-a666-0a27085f59a9" containerID="9b2d4e58f3155bada024cec69089e233d23b9ae375acbc12a861ac3312006b4f" exitCode=0
Sep 17 17:07:46.084611 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.084535    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8" event={"ID":"018c3162-3cfe-49d4-a666-0a27085f59a9","Type":"ContainerDied","Data":"9b2d4e58f3155bada024cec69089e233d23b9ae375acbc12a861ac3312006b4f"}
Sep 17 17:07:46.084611 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.084573    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8" event={"ID":"018c3162-3cfe-49d4-a666-0a27085f59a9","Type":"ContainerDied","Data":"2b8fd221b074f89ebb4f2e977db7e399c235cac2b3cdb8b3fcdd39770e20218e"}
Sep 17 17:07:46.084611 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.084576    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"
Sep 17 17:07:46.093097 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.093075    2581 scope.go:117] "RemoveContainer" containerID="70e9cf7ba69f3e98343f411387445dc8e377904702a918bcfd26a23a20eedbd4"
Sep 17 17:07:46.101789 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.101771    2581 scope.go:117] "RemoveContainer" containerID="6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8"
Sep 17 17:07:46.102101 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:07:46.102083    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8\": container with ID starting with 6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8 not found: ID does not exist" containerID="6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8"
Sep 17 17:07:46.102167 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.102109    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8"} err="failed to get container status \"6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8\": rpc error: code = NotFound desc = could not find container \"6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8\": container with ID starting with 6440f4be8ddbb7e34e5969a33f1bc5afe72f1d3fba11266cb886fe7b3848ceb8 not found: ID does not exist"
Sep 17 17:07:46.102167 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.102123    2581 scope.go:117] "RemoveContainer" containerID="70e9cf7ba69f3e98343f411387445dc8e377904702a918bcfd26a23a20eedbd4"
Sep 17 17:07:46.102397 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:07:46.102380    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e9cf7ba69f3e98343f411387445dc8e377904702a918bcfd26a23a20eedbd4\": container with ID starting with 70e9cf7ba69f3e98343f411387445dc8e377904702a918bcfd26a23a20eedbd4 not found: ID does not exist" containerID="70e9cf7ba69f3e98343f411387445dc8e377904702a918bcfd26a23a20eedbd4"
Sep 17 17:07:46.102435 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.102403    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e9cf7ba69f3e98343f411387445dc8e377904702a918bcfd26a23a20eedbd4"} err="failed to get container status \"70e9cf7ba69f3e98343f411387445dc8e377904702a918bcfd26a23a20eedbd4\": rpc error: code = NotFound desc = could not find container \"70e9cf7ba69f3e98343f411387445dc8e377904702a918bcfd26a23a20eedbd4\": container with ID starting with 70e9cf7ba69f3e98343f411387445dc8e377904702a918bcfd26a23a20eedbd4 not found: ID does not exist"
Sep 17 17:07:46.102435 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.102417    2581 scope.go:117] "RemoveContainer" containerID="9b2d4e58f3155bada024cec69089e233d23b9ae375acbc12a861ac3312006b4f"
Sep 17 17:07:46.109817 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.109799    2581 scope.go:117] "RemoveContainer" containerID="1783aa04dcec8e2e969e9b3337e67e8b514d45863e821f5b9abf7e6b377be297"
Sep 17 17:07:46.111021 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.110958    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"]
Sep 17 17:07:46.115717 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.115691    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c49c77444-ndz5d"]
Sep 17 17:07:46.117390 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.117376    2581 scope.go:117] "RemoveContainer" containerID="9b2d4e58f3155bada024cec69089e233d23b9ae375acbc12a861ac3312006b4f"
Sep 17 17:07:46.117651 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:07:46.117633    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2d4e58f3155bada024cec69089e233d23b9ae375acbc12a861ac3312006b4f\": container with ID starting with 9b2d4e58f3155bada024cec69089e233d23b9ae375acbc12a861ac3312006b4f not found: ID does not exist" containerID="9b2d4e58f3155bada024cec69089e233d23b9ae375acbc12a861ac3312006b4f"
Sep 17 17:07:46.117697 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.117657    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2d4e58f3155bada024cec69089e233d23b9ae375acbc12a861ac3312006b4f"} err="failed to get container status \"9b2d4e58f3155bada024cec69089e233d23b9ae375acbc12a861ac3312006b4f\": rpc error: code = NotFound desc = could not find container \"9b2d4e58f3155bada024cec69089e233d23b9ae375acbc12a861ac3312006b4f\": container with ID starting with 9b2d4e58f3155bada024cec69089e233d23b9ae375acbc12a861ac3312006b4f not found: ID does not exist"
Sep 17 17:07:46.117697 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.117672    2581 scope.go:117] "RemoveContainer" containerID="1783aa04dcec8e2e969e9b3337e67e8b514d45863e821f5b9abf7e6b377be297"
Sep 17 17:07:46.117925 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:07:46.117906    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1783aa04dcec8e2e969e9b3337e67e8b514d45863e821f5b9abf7e6b377be297\": container with ID starting with 1783aa04dcec8e2e969e9b3337e67e8b514d45863e821f5b9abf7e6b377be297 not found: ID does not exist" containerID="1783aa04dcec8e2e969e9b3337e67e8b514d45863e821f5b9abf7e6b377be297"
Sep 17 17:07:46.117981 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.117930    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1783aa04dcec8e2e969e9b3337e67e8b514d45863e821f5b9abf7e6b377be297"} err="failed to get container status \"1783aa04dcec8e2e969e9b3337e67e8b514d45863e821f5b9abf7e6b377be297\": rpc error: code = NotFound desc = could not find container \"1783aa04dcec8e2e969e9b3337e67e8b514d45863e821f5b9abf7e6b377be297\": container with ID starting with 1783aa04dcec8e2e969e9b3337e67e8b514d45863e821f5b9abf7e6b377be297 not found: ID does not exist"
Sep 17 17:07:46.129329 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.129309    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"]
Sep 17 17:07:46.137775 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.137755    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-78dcd6594b-2tsk8"]
Sep 17 17:07:46.836310 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.836278    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="018c3162-3cfe-49d4-a666-0a27085f59a9" path="/var/lib/kubelet/pods/018c3162-3cfe-49d4-a666-0a27085f59a9/volumes"
Sep 17 17:07:46.836741 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:07:46.836727    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20805d1d-8d83-4526-af1a-ca2b056f8bb9" path="/var/lib/kubelet/pods/20805d1d-8d83-4526-af1a-ca2b056f8bb9/volumes"
Sep 17 17:08:26.615219 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:26.615170    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"]
Sep 17 17:08:26.690784 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:26.690745    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965" podUID="a5afe50b-814b-4582-a81d-4d98c31d85fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:08:36.690505 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:36.690455    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965" podUID="a5afe50b-814b-4582-a81d-4d98c31d85fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:08:41.634548 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:41.634484    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965" podUID="a5afe50b-814b-4582-a81d-4d98c31d85fc" containerName="main" containerID="cri-o://1d495742d72645e675312b130412fb58233b0eefa5fe5d0f5e7b519c3895cb66" gracePeriod=45
Sep 17 17:08:46.421751 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.421677    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"]
Sep 17 17:08:46.422103 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.422063    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="018c3162-3cfe-49d4-a666-0a27085f59a9" containerName="storage-initializer"
Sep 17 17:08:46.422103 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.422077    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="018c3162-3cfe-49d4-a666-0a27085f59a9" containerName="storage-initializer"
Sep 17 17:08:46.422103 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.422085    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20805d1d-8d83-4526-af1a-ca2b056f8bb9" containerName="main"
Sep 17 17:08:46.422103 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.422091    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="20805d1d-8d83-4526-af1a-ca2b056f8bb9" containerName="main"
Sep 17 17:08:46.422103 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.422098    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="018c3162-3cfe-49d4-a666-0a27085f59a9" containerName="main"
Sep 17 17:08:46.422103 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.422103    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="018c3162-3cfe-49d4-a666-0a27085f59a9" containerName="main"
Sep 17 17:08:46.422349 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.422126    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20805d1d-8d83-4526-af1a-ca2b056f8bb9" containerName="storage-initializer"
Sep 17 17:08:46.422349 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.422131    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="20805d1d-8d83-4526-af1a-ca2b056f8bb9" containerName="storage-initializer"
Sep 17 17:08:46.422349 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.422199    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="20805d1d-8d83-4526-af1a-ca2b056f8bb9" containerName="main"
Sep 17 17:08:46.422349 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.422209    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="018c3162-3cfe-49d4-a666-0a27085f59a9" containerName="main"
Sep 17 17:08:46.425341 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.425324    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.428895 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.428872    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 17:08:46.429031 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.428901    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 17:08:46.429031 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.428911    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 17:08:46.429707 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.429684    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:08:46.429757 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.429707    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-fsnqb\""
Sep 17 17:08:46.429757 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.429713    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 17:08:46.439697 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.439675    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"]
Sep 17 17:08:46.555635 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.555608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.555768 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.555642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-tls-certs\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.555768 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.555666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvpfq\" (UniqueName: \"kubernetes.io/projected/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-kube-api-access-pvpfq\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.555898 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.555764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-model-cache\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.555898 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.555823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-dshm\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.555898 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.555870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.556067 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.555934    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-home\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.556067 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.555989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.656509 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.656482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.656613 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.656521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.656613 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.656546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-tls-certs\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.656703 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.656651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvpfq\" (UniqueName: \"kubernetes.io/projected/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-kube-api-access-pvpfq\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.656739 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.656709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-model-cache\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.656866 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.656847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.656924 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.656897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-dshm\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.656993 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.656925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.656993 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.656953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.657114 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.657017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-home\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.657114 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.657028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-model-cache\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.657325 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.657300    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-home\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.657702 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.657678    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.659250 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.659220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-dshm\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.659358 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.659343    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-tls-certs\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.668443 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.668421    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvpfq\" (UniqueName: \"kubernetes.io/projected/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-kube-api-access-pvpfq\") pod \"tokenizer-template-args-test-kserve-7bf85b5796-n4ks2\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.696529 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.693731    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965" podUID="a5afe50b-814b-4582-a81d-4d98c31d85fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:08:46.696529 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.693917    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"
Sep 17 17:08:46.701400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.701372    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"]
Sep 17 17:08:46.706051 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.706032    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:46.717695 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.717675    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"]
Sep 17 17:08:46.735909 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.735881    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:46.757854 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.757825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40851066-57ec-46fa-af9c-2528197267a3-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-f68f57854-8j55j\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:46.757975 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.757889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40851066-57ec-46fa-af9c-2528197267a3-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-f68f57854-8j55j\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:46.757975 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.757910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40851066-57ec-46fa-af9c-2528197267a3-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-f68f57854-8j55j\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:46.758108 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.758066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40851066-57ec-46fa-af9c-2528197267a3-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-f68f57854-8j55j\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:46.858461 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.858429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40851066-57ec-46fa-af9c-2528197267a3-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-f68f57854-8j55j\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:46.858461 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.858460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40851066-57ec-46fa-af9c-2528197267a3-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-f68f57854-8j55j\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:46.858646 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.858515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40851066-57ec-46fa-af9c-2528197267a3-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-f68f57854-8j55j\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:46.858646 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.858549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40851066-57ec-46fa-af9c-2528197267a3-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-f68f57854-8j55j\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:46.858903 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.858881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40851066-57ec-46fa-af9c-2528197267a3-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-f68f57854-8j55j\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:46.858943 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.858915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40851066-57ec-46fa-af9c-2528197267a3-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-f68f57854-8j55j\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:46.859207 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.859189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40851066-57ec-46fa-af9c-2528197267a3-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-f68f57854-8j55j\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:46.860894 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:46.860872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40851066-57ec-46fa-af9c-2528197267a3-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-f68f57854-8j55j\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:47.015425 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:47.015406    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"]
Sep 17 17:08:47.017556 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:47.017536    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:47.018102 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:08:47.018081    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0c33f3_1ebb_4852_a239_8e688ded3bb6.slice/crio-be78ecbf3d03d87fed37a88185e49a07f0ea8b3a79522eb8af36cb3c1739c360 WatchSource:0}: Error finding container be78ecbf3d03d87fed37a88185e49a07f0ea8b3a79522eb8af36cb3c1739c360: Status 404 returned error can't find the container with id be78ecbf3d03d87fed37a88185e49a07f0ea8b3a79522eb8af36cb3c1739c360
Sep 17 17:08:47.302709 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:47.302683    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"]
Sep 17 17:08:47.305364 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:08:47.305336    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40851066_57ec_46fa_af9c_2528197267a3.slice/crio-e8aa2b32769033e4d5cad1cd3639dfa91ab54f5f9884c2d94907d279e4b4eba2 WatchSource:0}: Error finding container e8aa2b32769033e4d5cad1cd3639dfa91ab54f5f9884c2d94907d279e4b4eba2: Status 404 returned error can't find the container with id e8aa2b32769033e4d5cad1cd3639dfa91ab54f5f9884c2d94907d279e4b4eba2
Sep 17 17:08:47.379545 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:47.379514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j" event={"ID":"40851066-57ec-46fa-af9c-2528197267a3","Type":"ContainerStarted","Data":"e8aa2b32769033e4d5cad1cd3639dfa91ab54f5f9884c2d94907d279e4b4eba2"}
Sep 17 17:08:47.381231 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:47.381191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2" event={"ID":"8b0c33f3-1ebb-4852-a239-8e688ded3bb6","Type":"ContainerStarted","Data":"08670e4e285e2264cef0b0760b9f3a5007aaaa1e5b418852ba8e322c2728d6ba"}
Sep 17 17:08:47.381351 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:47.381247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2" event={"ID":"8b0c33f3-1ebb-4852-a239-8e688ded3bb6","Type":"ContainerStarted","Data":"be78ecbf3d03d87fed37a88185e49a07f0ea8b3a79522eb8af36cb3c1739c360"}
Sep 17 17:08:48.386763 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:48.386725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j" event={"ID":"40851066-57ec-46fa-af9c-2528197267a3","Type":"ContainerStarted","Data":"c3b927c0373b9efc5765468e43d058da3d468356d923a19381a719d2a337fabc"}
Sep 17 17:08:50.398732 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:50.398697    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b0c33f3-1ebb-4852-a239-8e688ded3bb6" containerID="08670e4e285e2264cef0b0760b9f3a5007aaaa1e5b418852ba8e322c2728d6ba" exitCode=0
Sep 17 17:08:50.399246 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:50.398784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2" event={"ID":"8b0c33f3-1ebb-4852-a239-8e688ded3bb6","Type":"ContainerDied","Data":"08670e4e285e2264cef0b0760b9f3a5007aaaa1e5b418852ba8e322c2728d6ba"}
Sep 17 17:08:50.400754 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:50.400730    2581 generic.go:356] "Generic (PLEG): container finished" podID="40851066-57ec-46fa-af9c-2528197267a3" containerID="c3b927c0373b9efc5765468e43d058da3d468356d923a19381a719d2a337fabc" exitCode=0
Sep 17 17:08:50.400860 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:50.400804    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j" event={"ID":"40851066-57ec-46fa-af9c-2528197267a3","Type":"ContainerDied","Data":"c3b927c0373b9efc5765468e43d058da3d468356d923a19381a719d2a337fabc"}
Sep 17 17:08:51.406649 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:51.406608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2" event={"ID":"8b0c33f3-1ebb-4852-a239-8e688ded3bb6","Type":"ContainerStarted","Data":"2322b3ed9d5d160776dd34ac2ecfb7e7e7356b036417cb8864551802e00949ab"}
Sep 17 17:08:51.408525 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:51.408498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j" event={"ID":"40851066-57ec-46fa-af9c-2528197267a3","Type":"ContainerStarted","Data":"ff82bc66cff0acda6e3fbfce71c9846482d0edce46b900d331de66f96abae98b"}
Sep 17 17:08:51.408707 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:51.408692    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:08:51.435132 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:51.435089    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2" podStartSLOduration=3.008781313 podStartE2EDuration="5.435076645s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.426295332s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:08:51.433197093 +0000 UTC m=+1610.856134955" watchObservedRunningTime="2026-09-17 17:08:51.435076645 +0000 UTC m=+1610.858014508"
Sep 17 17:08:51.457323 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:51.457281    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j" podStartSLOduration=3.046315738 podStartE2EDuration="5.457267971s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.410952233s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:08:51.455799931 +0000 UTC m=+1610.878737794" watchObservedRunningTime="2026-09-17 17:08:51.457267971 +0000 UTC m=+1610.880205860"
Sep 17 17:08:51.736917 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:51.736888    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:56.690777 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:56.690741    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965" podUID="a5afe50b-814b-4582-a81d-4d98c31d85fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:08:56.736322 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:56.736285    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:56.748657 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:56.748634    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:08:57.452019 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:08:57.451987    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:09:06.690586 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:06.690538    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965" podUID="a5afe50b-814b-4582-a81d-4d98c31d85fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:09:11.864427 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:11.864397    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"
Sep 17 17:09:11.968622 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:11.968596    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5afe50b-814b-4582-a81d-4d98c31d85fc-tls-certs\") pod \"a5afe50b-814b-4582-a81d-4d98c31d85fc\" (UID: \"a5afe50b-814b-4582-a81d-4d98c31d85fc\") "
Sep 17 17:09:11.968755 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:11.968628    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5hpv\" (UniqueName: \"kubernetes.io/projected/a5afe50b-814b-4582-a81d-4d98c31d85fc-kube-api-access-c5hpv\") pod \"a5afe50b-814b-4582-a81d-4d98c31d85fc\" (UID: \"a5afe50b-814b-4582-a81d-4d98c31d85fc\") "
Sep 17 17:09:11.970699 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:11.970669    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5afe50b-814b-4582-a81d-4d98c31d85fc-kube-api-access-c5hpv" (OuterVolumeSpecName: "kube-api-access-c5hpv") pod "a5afe50b-814b-4582-a81d-4d98c31d85fc" (UID: "a5afe50b-814b-4582-a81d-4d98c31d85fc"). InnerVolumeSpecName "kube-api-access-c5hpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:09:11.970809 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:11.970751    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5afe50b-814b-4582-a81d-4d98c31d85fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5afe50b-814b-4582-a81d-4d98c31d85fc" (UID: "a5afe50b-814b-4582-a81d-4d98c31d85fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:09:12.069644 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:12.069620    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5afe50b-814b-4582-a81d-4d98c31d85fc-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:09:12.069644 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:12.069641    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5hpv\" (UniqueName: \"kubernetes.io/projected/a5afe50b-814b-4582-a81d-4d98c31d85fc-kube-api-access-c5hpv\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:09:12.513102 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:12.513074    2581 generic.go:356] "Generic (PLEG): container finished" podID="a5afe50b-814b-4582-a81d-4d98c31d85fc" containerID="1d495742d72645e675312b130412fb58233b0eefa5fe5d0f5e7b519c3895cb66" exitCode=0
Sep 17 17:09:12.513230 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:12.513116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965" event={"ID":"a5afe50b-814b-4582-a81d-4d98c31d85fc","Type":"ContainerDied","Data":"1d495742d72645e675312b130412fb58233b0eefa5fe5d0f5e7b519c3895cb66"}
Sep 17 17:09:12.513230 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:12.513123    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"
Sep 17 17:09:12.513230 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:12.513165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965" event={"ID":"a5afe50b-814b-4582-a81d-4d98c31d85fc","Type":"ContainerDied","Data":"01fb1527ef25df4062b50e9da10da086419db53ae73206f423c511cadae48527"}
Sep 17 17:09:12.513230 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:12.513188    2581 scope.go:117] "RemoveContainer" containerID="1d495742d72645e675312b130412fb58233b0eefa5fe5d0f5e7b519c3895cb66"
Sep 17 17:09:12.522349 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:12.522325    2581 scope.go:117] "RemoveContainer" containerID="1d495742d72645e675312b130412fb58233b0eefa5fe5d0f5e7b519c3895cb66"
Sep 17 17:09:12.522631 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:09:12.522603    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d495742d72645e675312b130412fb58233b0eefa5fe5d0f5e7b519c3895cb66\": container with ID starting with 1d495742d72645e675312b130412fb58233b0eefa5fe5d0f5e7b519c3895cb66 not found: ID does not exist" containerID="1d495742d72645e675312b130412fb58233b0eefa5fe5d0f5e7b519c3895cb66"
Sep 17 17:09:12.522705 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:12.522642    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d495742d72645e675312b130412fb58233b0eefa5fe5d0f5e7b519c3895cb66"} err="failed to get container status \"1d495742d72645e675312b130412fb58233b0eefa5fe5d0f5e7b519c3895cb66\": rpc error: code = NotFound desc = could not find container \"1d495742d72645e675312b130412fb58233b0eefa5fe5d0f5e7b519c3895cb66\": container with ID starting with 1d495742d72645e675312b130412fb58233b0eefa5fe5d0f5e7b519c3895cb66 not found: ID does not exist"
Sep 17 17:09:12.537620 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:12.537594    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"]
Sep 17 17:09:12.544789 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:12.544761    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9clr965"]
Sep 17 17:09:12.836017 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:12.835932    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5afe50b-814b-4582-a81d-4d98c31d85fc" path="/var/lib/kubelet/pods/a5afe50b-814b-4582-a81d-4d98c31d85fc/volumes"
Sep 17 17:09:22.423695 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:22.423655    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:09:45.135420 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:45.135375    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-f68f57854-8j55j_40851066-57ec-46fa-af9c-2528197267a3/main/0.log"
Sep 17 17:09:47.490242 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.490212    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"]
Sep 17 17:09:47.499157 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:09:47.499123    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:09:47.499157 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:09:47.499157 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:09:47.499157 ip-10-0-140-49 kubenswrapper[2581]:  > containerID="2322b3ed9d5d160776dd34ac2ecfb7e7e7356b036417cb8864551802e00949ab" cmd=["/bin/sleep","15"]
Sep 17 17:09:47.499442 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.499176    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2" message="" err=<
Sep 17 17:09:47.499442 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:09:47.499442 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:09:47.499442 ip-10-0-140-49 kubenswrapper[2581]:  >
Sep 17 17:09:47.499442 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:09:47.499196    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:09:47.499442 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:09:47.499442 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:09:47.499442 ip-10-0-140-49 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2" podUID="8b0c33f3-1ebb-4852-a239-8e688ded3bb6" containerName="main" containerID="cri-o://2322b3ed9d5d160776dd34ac2ecfb7e7e7356b036417cb8864551802e00949ab"
Sep 17 17:09:47.499442 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.499237    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2" podUID="8b0c33f3-1ebb-4852-a239-8e688ded3bb6" containerName="main" containerID="cri-o://2322b3ed9d5d160776dd34ac2ecfb7e7e7356b036417cb8864551802e00949ab" gracePeriod=60
Sep 17 17:09:47.499988 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.499909    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"]
Sep 17 17:09:47.500121 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.500103    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j" podUID="40851066-57ec-46fa-af9c-2528197267a3" containerName="main" containerID="cri-o://ff82bc66cff0acda6e3fbfce71c9846482d0edce46b900d331de66f96abae98b" gracePeriod=30
Sep 17 17:09:47.690913 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.690882    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b0c33f3-1ebb-4852-a239-8e688ded3bb6" containerID="2322b3ed9d5d160776dd34ac2ecfb7e7e7356b036417cb8864551802e00949ab" exitCode=0
Sep 17 17:09:47.691090 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.690981    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2" event={"ID":"8b0c33f3-1ebb-4852-a239-8e688ded3bb6","Type":"ContainerDied","Data":"2322b3ed9d5d160776dd34ac2ecfb7e7e7356b036417cb8864551802e00949ab"}
Sep 17 17:09:47.747439 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.747386    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:09:47.827181 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-cabundle-cert\") pod \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") "
Sep 17 17:09:47.827365 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827215    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-dshm\") pod \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") "
Sep 17 17:09:47.827365 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827233    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-home\") pod \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") "
Sep 17 17:09:47.827365 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827251    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvpfq\" (UniqueName: \"kubernetes.io/projected/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-kube-api-access-pvpfq\") pod \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") "
Sep 17 17:09:47.827551 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827417    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-tmp-dir\") pod \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") "
Sep 17 17:09:47.827551 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827483    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-home" (OuterVolumeSpecName: "home") pod "8b0c33f3-1ebb-4852-a239-8e688ded3bb6" (UID: "8b0c33f3-1ebb-4852-a239-8e688ded3bb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:47.827551 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827494    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-tls-certs\") pod \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") "
Sep 17 17:09:47.827551 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827533    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b0c33f3-1ebb-4852-a239-8e688ded3bb6" (UID: "8b0c33f3-1ebb-4852-a239-8e688ded3bb6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:09:47.827766 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827540    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-kserve-provision-location\") pod \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") "
Sep 17 17:09:47.827766 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827614    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-model-cache\") pod \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\" (UID: \"8b0c33f3-1ebb-4852-a239-8e688ded3bb6\") "
Sep 17 17:09:47.827766 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827621    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b0c33f3-1ebb-4852-a239-8e688ded3bb6" (UID: "8b0c33f3-1ebb-4852-a239-8e688ded3bb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:47.827926 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827880    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:09:47.827926 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827884    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b0c33f3-1ebb-4852-a239-8e688ded3bb6" (UID: "8b0c33f3-1ebb-4852-a239-8e688ded3bb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:47.827926 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827894    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:09:47.828131 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.827925    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:09:47.829564 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.829536    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-dshm" (OuterVolumeSpecName: "dshm") pod "8b0c33f3-1ebb-4852-a239-8e688ded3bb6" (UID: "8b0c33f3-1ebb-4852-a239-8e688ded3bb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:47.829694 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.829650    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-kube-api-access-pvpfq" (OuterVolumeSpecName: "kube-api-access-pvpfq") pod "8b0c33f3-1ebb-4852-a239-8e688ded3bb6" (UID: "8b0c33f3-1ebb-4852-a239-8e688ded3bb6"). InnerVolumeSpecName "kube-api-access-pvpfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:09:47.829807 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.829790    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b0c33f3-1ebb-4852-a239-8e688ded3bb6" (UID: "8b0c33f3-1ebb-4852-a239-8e688ded3bb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:09:47.850553 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.850518    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b0c33f3-1ebb-4852-a239-8e688ded3bb6" (UID: "8b0c33f3-1ebb-4852-a239-8e688ded3bb6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:47.929562 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.929535    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvpfq\" (UniqueName: \"kubernetes.io/projected/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-kube-api-access-pvpfq\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:09:47.929666 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.929570    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:09:47.929666 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.929589    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:09:47.929666 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.929607    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:09:47.929666 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:47.929623    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0c33f3-1ebb-4852-a239-8e688ded3bb6-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:09:48.040334 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.040308    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:09:48.131038 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.131012    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40851066-57ec-46fa-af9c-2528197267a3-kserve-provision-location\") pod \"40851066-57ec-46fa-af9c-2528197267a3\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") "
Sep 17 17:09:48.131180 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.131081    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40851066-57ec-46fa-af9c-2528197267a3-tls-certs\") pod \"40851066-57ec-46fa-af9c-2528197267a3\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") "
Sep 17 17:09:48.131180 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.131109    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40851066-57ec-46fa-af9c-2528197267a3-tmp-dir\") pod \"40851066-57ec-46fa-af9c-2528197267a3\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") "
Sep 17 17:09:48.131180 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.131172    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40851066-57ec-46fa-af9c-2528197267a3-cabundle-cert\") pod \"40851066-57ec-46fa-af9c-2528197267a3\" (UID: \"40851066-57ec-46fa-af9c-2528197267a3\") "
Sep 17 17:09:48.131561 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.131540    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40851066-57ec-46fa-af9c-2528197267a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40851066-57ec-46fa-af9c-2528197267a3" (UID: "40851066-57ec-46fa-af9c-2528197267a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:48.131626 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.131566    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40851066-57ec-46fa-af9c-2528197267a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "40851066-57ec-46fa-af9c-2528197267a3" (UID: "40851066-57ec-46fa-af9c-2528197267a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:09:48.133171 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.133150    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40851066-57ec-46fa-af9c-2528197267a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40851066-57ec-46fa-af9c-2528197267a3" (UID: "40851066-57ec-46fa-af9c-2528197267a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:09:48.152770 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.152742    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40851066-57ec-46fa-af9c-2528197267a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40851066-57ec-46fa-af9c-2528197267a3" (UID: "40851066-57ec-46fa-af9c-2528197267a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:48.231902 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.231882    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40851066-57ec-46fa-af9c-2528197267a3-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:09:48.231902 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.231901    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40851066-57ec-46fa-af9c-2528197267a3-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:09:48.232089 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.231912    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40851066-57ec-46fa-af9c-2528197267a3-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:09:48.232089 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.231921    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40851066-57ec-46fa-af9c-2528197267a3-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:09:48.698384 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.698359    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"
Sep 17 17:09:48.698765 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.698389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2" event={"ID":"8b0c33f3-1ebb-4852-a239-8e688ded3bb6","Type":"ContainerDied","Data":"be78ecbf3d03d87fed37a88185e49a07f0ea8b3a79522eb8af36cb3c1739c360"}
Sep 17 17:09:48.698765 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.698465    2581 scope.go:117] "RemoveContainer" containerID="2322b3ed9d5d160776dd34ac2ecfb7e7e7356b036417cb8864551802e00949ab"
Sep 17 17:09:48.700106 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.700080    2581 generic.go:356] "Generic (PLEG): container finished" podID="40851066-57ec-46fa-af9c-2528197267a3" containerID="ff82bc66cff0acda6e3fbfce71c9846482d0edce46b900d331de66f96abae98b" exitCode=0
Sep 17 17:09:48.700283 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.700135    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"
Sep 17 17:09:48.700283 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.700159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j" event={"ID":"40851066-57ec-46fa-af9c-2528197267a3","Type":"ContainerDied","Data":"ff82bc66cff0acda6e3fbfce71c9846482d0edce46b900d331de66f96abae98b"}
Sep 17 17:09:48.700283 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.700201    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j" event={"ID":"40851066-57ec-46fa-af9c-2528197267a3","Type":"ContainerDied","Data":"e8aa2b32769033e4d5cad1cd3639dfa91ab54f5f9884c2d94907d279e4b4eba2"}
Sep 17 17:09:48.707851 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.707745    2581 scope.go:117] "RemoveContainer" containerID="08670e4e285e2264cef0b0760b9f3a5007aaaa1e5b418852ba8e322c2728d6ba"
Sep 17 17:09:48.715811 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.715796    2581 scope.go:117] "RemoveContainer" containerID="ff82bc66cff0acda6e3fbfce71c9846482d0edce46b900d331de66f96abae98b"
Sep 17 17:09:48.723040 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.723021    2581 scope.go:117] "RemoveContainer" containerID="c3b927c0373b9efc5765468e43d058da3d468356d923a19381a719d2a337fabc"
Sep 17 17:09:48.724853 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.724835    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"]
Sep 17 17:09:48.731189 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.731169    2581 scope.go:117] "RemoveContainer" containerID="ff82bc66cff0acda6e3fbfce71c9846482d0edce46b900d331de66f96abae98b"
Sep 17 17:09:48.731842 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.731814    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-f68f57854-8j55j"]
Sep 17 17:09:48.731929 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:09:48.731865    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff82bc66cff0acda6e3fbfce71c9846482d0edce46b900d331de66f96abae98b\": container with ID starting with ff82bc66cff0acda6e3fbfce71c9846482d0edce46b900d331de66f96abae98b not found: ID does not exist" containerID="ff82bc66cff0acda6e3fbfce71c9846482d0edce46b900d331de66f96abae98b"
Sep 17 17:09:48.731929 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.731898    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff82bc66cff0acda6e3fbfce71c9846482d0edce46b900d331de66f96abae98b"} err="failed to get container status \"ff82bc66cff0acda6e3fbfce71c9846482d0edce46b900d331de66f96abae98b\": rpc error: code = NotFound desc = could not find container \"ff82bc66cff0acda6e3fbfce71c9846482d0edce46b900d331de66f96abae98b\": container with ID starting with ff82bc66cff0acda6e3fbfce71c9846482d0edce46b900d331de66f96abae98b not found: ID does not exist"
Sep 17 17:09:48.731929 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.731922    2581 scope.go:117] "RemoveContainer" containerID="c3b927c0373b9efc5765468e43d058da3d468356d923a19381a719d2a337fabc"
Sep 17 17:09:48.732247 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:09:48.732226    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b927c0373b9efc5765468e43d058da3d468356d923a19381a719d2a337fabc\": container with ID starting with c3b927c0373b9efc5765468e43d058da3d468356d923a19381a719d2a337fabc not found: ID does not exist" containerID="c3b927c0373b9efc5765468e43d058da3d468356d923a19381a719d2a337fabc"
Sep 17 17:09:48.732304 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.732256    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b927c0373b9efc5765468e43d058da3d468356d923a19381a719d2a337fabc"} err="failed to get container status \"c3b927c0373b9efc5765468e43d058da3d468356d923a19381a719d2a337fabc\": rpc error: code = NotFound desc = could not find container \"c3b927c0373b9efc5765468e43d058da3d468356d923a19381a719d2a337fabc\": container with ID starting with c3b927c0373b9efc5765468e43d058da3d468356d923a19381a719d2a337fabc not found: ID does not exist"
Sep 17 17:09:48.745370 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.745346    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"]
Sep 17 17:09:48.750658 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.750632    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7bf85b5796-n4ks2"]
Sep 17 17:09:48.835294 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.835269    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40851066-57ec-46fa-af9c-2528197267a3" path="/var/lib/kubelet/pods/40851066-57ec-46fa-af9c-2528197267a3/volumes"
Sep 17 17:09:48.835703 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:09:48.835690    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b0c33f3-1ebb-4852-a239-8e688ded3bb6" path="/var/lib/kubelet/pods/8b0c33f3-1ebb-4852-a239-8e688ded3bb6/volumes"
Sep 17 17:11:01.486946 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.486904    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"]
Sep 17 17:11:01.487541 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.487263    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5afe50b-814b-4582-a81d-4d98c31d85fc" containerName="main"
Sep 17 17:11:01.487541 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.487276    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5afe50b-814b-4582-a81d-4d98c31d85fc" containerName="main"
Sep 17 17:11:01.487541 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.487285    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40851066-57ec-46fa-af9c-2528197267a3" containerName="storage-initializer"
Sep 17 17:11:01.487541 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.487291    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="40851066-57ec-46fa-af9c-2528197267a3" containerName="storage-initializer"
Sep 17 17:11:01.487541 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.487304    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b0c33f3-1ebb-4852-a239-8e688ded3bb6" containerName="storage-initializer"
Sep 17 17:11:01.487541 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.487311    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b0c33f3-1ebb-4852-a239-8e688ded3bb6" containerName="storage-initializer"
Sep 17 17:11:01.487541 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.487318    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40851066-57ec-46fa-af9c-2528197267a3" containerName="main"
Sep 17 17:11:01.487541 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.487326    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="40851066-57ec-46fa-af9c-2528197267a3" containerName="main"
Sep 17 17:11:01.487541 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.487340    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b0c33f3-1ebb-4852-a239-8e688ded3bb6" containerName="main"
Sep 17 17:11:01.487541 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.487345    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b0c33f3-1ebb-4852-a239-8e688ded3bb6" containerName="main"
Sep 17 17:11:01.487541 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.487407    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b0c33f3-1ebb-4852-a239-8e688ded3bb6" containerName="main"
Sep 17 17:11:01.487541 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.487416    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5afe50b-814b-4582-a81d-4d98c31d85fc" containerName="main"
Sep 17 17:11:01.487541 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.487422    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="40851066-57ec-46fa-af9c-2528197267a3" containerName="main"
Sep 17 17:11:01.490449 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.490430    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.494183 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.494149    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 17:11:01.494350 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.494196    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 17:11:01.494350 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.494232    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-d88dz\""
Sep 17 17:11:01.494350 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.494235    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 17:11:01.504312 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.504283    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"]
Sep 17 17:11:01.574867 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.574831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.575037 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.574879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2hms\" (UniqueName: \"kubernetes.io/projected/13dcd858-9e73-4410-9224-15df6fdc36c8-kube-api-access-k2hms\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.575037 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.574902    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.575037 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.574921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.575037 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.574936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13dcd858-9e73-4410-9224-15df6fdc36c8-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.575037 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.575019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.675326 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.675299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.675455 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.675340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hms\" (UniqueName: \"kubernetes.io/projected/13dcd858-9e73-4410-9224-15df6fdc36c8-kube-api-access-k2hms\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.675455 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.675363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.675455 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.675380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.675623 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.675506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13dcd858-9e73-4410-9224-15df6fdc36c8-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.675623 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.675598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.675720 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.675702    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.675813 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.675790    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.675855 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.675820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.677891 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.677862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.678161 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.678142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13dcd858-9e73-4410-9224-15df6fdc36c8-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.684915 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.684887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hms\" (UniqueName: \"kubernetes.io/projected/13dcd858-9e73-4410-9224-15df6fdc36c8-kube-api-access-k2hms\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:01.801461 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:01.801399    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:02.081015 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:02.080991    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"]
Sep 17 17:11:02.083554 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:11:02.083527    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13dcd858_9e73_4410_9224_15df6fdc36c8.slice/crio-2e919ea4f89dc7b891cac89bcbe40290b31d7e6b3dce8fed8505f3830ab9d611 WatchSource:0}: Error finding container 2e919ea4f89dc7b891cac89bcbe40290b31d7e6b3dce8fed8505f3830ab9d611: Status 404 returned error can't find the container with id 2e919ea4f89dc7b891cac89bcbe40290b31d7e6b3dce8fed8505f3830ab9d611
Sep 17 17:11:03.052347 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:03.052311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr" event={"ID":"13dcd858-9e73-4410-9224-15df6fdc36c8","Type":"ContainerStarted","Data":"d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7"}
Sep 17 17:11:03.052704 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:03.052362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr" event={"ID":"13dcd858-9e73-4410-9224-15df6fdc36c8","Type":"ContainerStarted","Data":"2e919ea4f89dc7b891cac89bcbe40290b31d7e6b3dce8fed8505f3830ab9d611"}
Sep 17 17:11:03.074115 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:03.074066    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr" podStartSLOduration=2.074053392 podStartE2EDuration="2.074053392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:11:03.073066017 +0000 UTC m=+1742.496003892" watchObservedRunningTime="2026-09-17 17:11:03.074053392 +0000 UTC m=+1742.496991254"
Sep 17 17:11:03.802188 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:03.802155    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:11.802084 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:11.801989    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:11.815370 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:11.815345    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:12.106310 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:12.106222    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:35.145604 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:35.145573    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"]
Sep 17 17:11:35.154676 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:11:35.154633    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:11:35.154676 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:11:35.154676 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:11:35.154676 ip-10-0-140-49 kubenswrapper[2581]:  > containerID="d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7" cmd=["/bin/sleep","15"]
Sep 17 17:11:35.155001 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:35.154696    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr" message="" err=<
Sep 17 17:11:35.155001 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:11:35.155001 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:11:35.155001 ip-10-0-140-49 kubenswrapper[2581]:  >
Sep 17 17:11:35.155001 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:11:35.154713    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:11:35.155001 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:11:35.155001 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:11:35.155001 ip-10-0-140-49 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr" podUID="13dcd858-9e73-4410-9224-15df6fdc36c8" containerName="main" containerID="cri-o://d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7"
Sep 17 17:11:35.155001 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:35.154759    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr" podUID="13dcd858-9e73-4410-9224-15df6fdc36c8" containerName="main" containerID="cri-o://d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7" gracePeriod=60
Sep 17 17:11:35.902595 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:35.902573    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:36.021072 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.021010    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-tmp-dir\") pod \"13dcd858-9e73-4410-9224-15df6fdc36c8\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") "
Sep 17 17:11:36.021072 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.021050    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2hms\" (UniqueName: \"kubernetes.io/projected/13dcd858-9e73-4410-9224-15df6fdc36c8-kube-api-access-k2hms\") pod \"13dcd858-9e73-4410-9224-15df6fdc36c8\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") "
Sep 17 17:11:36.021228 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.021078    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-home\") pod \"13dcd858-9e73-4410-9224-15df6fdc36c8\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") "
Sep 17 17:11:36.021228 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.021098    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-model-cache\") pod \"13dcd858-9e73-4410-9224-15df6fdc36c8\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") "
Sep 17 17:11:36.021228 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.021124    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13dcd858-9e73-4410-9224-15df6fdc36c8-tls-certs\") pod \"13dcd858-9e73-4410-9224-15df6fdc36c8\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") "
Sep 17 17:11:36.021228 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.021143    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-dshm\") pod \"13dcd858-9e73-4410-9224-15df6fdc36c8\" (UID: \"13dcd858-9e73-4410-9224-15df6fdc36c8\") "
Sep 17 17:11:36.021456 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.021295    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13dcd858-9e73-4410-9224-15df6fdc36c8" (UID: "13dcd858-9e73-4410-9224-15df6fdc36c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:11:36.021456 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.021323    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "13dcd858-9e73-4410-9224-15df6fdc36c8" (UID: "13dcd858-9e73-4410-9224-15df6fdc36c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:11:36.021456 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.021372    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-home" (OuterVolumeSpecName: "home") pod "13dcd858-9e73-4410-9224-15df6fdc36c8" (UID: "13dcd858-9e73-4410-9224-15df6fdc36c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:11:36.023211 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.023186    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-dshm" (OuterVolumeSpecName: "dshm") pod "13dcd858-9e73-4410-9224-15df6fdc36c8" (UID: "13dcd858-9e73-4410-9224-15df6fdc36c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:11:36.023289 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.023237    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13dcd858-9e73-4410-9224-15df6fdc36c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13dcd858-9e73-4410-9224-15df6fdc36c8" (UID: "13dcd858-9e73-4410-9224-15df6fdc36c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:11:36.023289 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.023265    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13dcd858-9e73-4410-9224-15df6fdc36c8-kube-api-access-k2hms" (OuterVolumeSpecName: "kube-api-access-k2hms") pod "13dcd858-9e73-4410-9224-15df6fdc36c8" (UID: "13dcd858-9e73-4410-9224-15df6fdc36c8"). InnerVolumeSpecName "kube-api-access-k2hms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:11:36.121761 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.121739    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:11:36.121761 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.121759    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2hms\" (UniqueName: \"kubernetes.io/projected/13dcd858-9e73-4410-9224-15df6fdc36c8-kube-api-access-k2hms\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:11:36.121892 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.121768    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:11:36.121892 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.121776    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:11:36.121892 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.121784    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13dcd858-9e73-4410-9224-15df6fdc36c8-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:11:36.121892 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.121790    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dcd858-9e73-4410-9224-15df6fdc36c8-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:11:36.205820 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.205794    2581 generic.go:356] "Generic (PLEG): container finished" podID="13dcd858-9e73-4410-9224-15df6fdc36c8" containerID="d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7" exitCode=0
Sep 17 17:11:36.206211 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.205832    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"
Sep 17 17:11:36.206211 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.205893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr" event={"ID":"13dcd858-9e73-4410-9224-15df6fdc36c8","Type":"ContainerDied","Data":"d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7"}
Sep 17 17:11:36.206211 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.205961    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr" event={"ID":"13dcd858-9e73-4410-9224-15df6fdc36c8","Type":"ContainerDied","Data":"2e919ea4f89dc7b891cac89bcbe40290b31d7e6b3dce8fed8505f3830ab9d611"}
Sep 17 17:11:36.206211 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.206011    2581 scope.go:117] "RemoveContainer" containerID="d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7"
Sep 17 17:11:36.215330 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.215309    2581 scope.go:117] "RemoveContainer" containerID="d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7"
Sep 17 17:11:36.215605 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:11:36.215587    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7\": container with ID starting with d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7 not found: ID does not exist" containerID="d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7"
Sep 17 17:11:36.215680 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.215613    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7"} err="failed to get container status \"d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7\": rpc error: code = NotFound desc = could not find container \"d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7\": container with ID starting with d7b5198183ae8b77da95140a1b1ac6eacc31125cd6fbb036aa4c313d54ac92d7 not found: ID does not exist"
Sep 17 17:11:36.229157 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.229137    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"]
Sep 17 17:11:36.235408 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.235388    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-zzcrr"]
Sep 17 17:11:36.837014 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:11:36.836961    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13dcd858-9e73-4410-9224-15df6fdc36c8" path="/var/lib/kubelet/pods/13dcd858-9e73-4410-9224-15df6fdc36c8/volumes"
Sep 17 17:12:02.850391 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:02.850362    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:12:02.853333 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:02.853306    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:12:30.171407 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.171373    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"]
Sep 17 17:12:30.171819 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.171692    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13dcd858-9e73-4410-9224-15df6fdc36c8" containerName="main"
Sep 17 17:12:30.171819 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.171704    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="13dcd858-9e73-4410-9224-15df6fdc36c8" containerName="main"
Sep 17 17:12:30.171819 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.171775    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="13dcd858-9e73-4410-9224-15df6fdc36c8" containerName="main"
Sep 17 17:12:30.174726 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.174706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.178085 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.178063    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 17:12:30.178197 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.178068    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 17 17:12:30.178257 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.178190    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-ktmh2\""
Sep 17 17:12:30.178257 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.178236    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 17:12:30.183470 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.183446    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"]
Sep 17 17:12:30.228727 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.228704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.228842 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.228738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wtlc\" (UniqueName: \"kubernetes.io/projected/7899f2bb-e5b5-4d0f-bc87-490af33ea684-kube-api-access-8wtlc\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.228842 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.228768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.228999 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.228846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7899f2bb-e5b5-4d0f-bc87-490af33ea684-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.228999 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.228911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.229109 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.229026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.330246 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.330221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.330369 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.330257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7899f2bb-e5b5-4d0f-bc87-490af33ea684-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.330369 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.330288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.330369 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.330337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.330369 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.330352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.330593 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.330372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wtlc\" (UniqueName: \"kubernetes.io/projected/7899f2bb-e5b5-4d0f-bc87-490af33ea684-kube-api-access-8wtlc\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.330677 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.330627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.330749 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.330681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.330749 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.330735    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.332531 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.332514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.332806 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.332787    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7899f2bb-e5b5-4d0f-bc87-490af33ea684-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.341582 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.341556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wtlc\" (UniqueName: \"kubernetes.io/projected/7899f2bb-e5b5-4d0f-bc87-490af33ea684-kube-api-access-8wtlc\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.486278 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.486254    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:30.762352 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:30.762328    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"]
Sep 17 17:12:30.764497 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:12:30.764469    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7899f2bb_e5b5_4d0f_bc87_490af33ea684.slice/crio-8273c252035852792bb3b687c9f091c06e951530df0bf2aeb0562d1891b1ec29 WatchSource:0}: Error finding container 8273c252035852792bb3b687c9f091c06e951530df0bf2aeb0562d1891b1ec29: Status 404 returned error can't find the container with id 8273c252035852792bb3b687c9f091c06e951530df0bf2aeb0562d1891b1ec29
Sep 17 17:12:31.467758 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:31.467725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg" event={"ID":"7899f2bb-e5b5-4d0f-bc87-490af33ea684","Type":"ContainerStarted","Data":"7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855"}
Sep 17 17:12:31.468130 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:31.467802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg" event={"ID":"7899f2bb-e5b5-4d0f-bc87-490af33ea684","Type":"ContainerStarted","Data":"8273c252035852792bb3b687c9f091c06e951530df0bf2aeb0562d1891b1ec29"}
Sep 17 17:12:31.486627 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:31.486597    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:31.489754 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:31.489710    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg" podStartSLOduration=1.489697756 podStartE2EDuration="1.489697756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:12:31.488350187 +0000 UTC m=+1830.911288050" watchObservedRunningTime="2026-09-17 17:12:31.489697756 +0000 UTC m=+1830.912635618"
Sep 17 17:12:40.486920 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:40.486884    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:40.499417 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:40.499340    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:12:40.521356 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:12:40.521333    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:13:44.534147 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.534112    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"]
Sep 17 17:13:44.537507 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.537487    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.551913 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.551893    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-vhr9x\""
Sep 17 17:13:44.552849 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.552829    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 17:13:44.552946 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.552918    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 17:13:44.553529 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.553512    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 17:13:44.561139 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.561106    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"]
Sep 17 17:13:44.610470 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.610446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.610590 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.610486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.610590 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.610504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.610702 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.610583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.610702 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.610626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.610702 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.610649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs48t\" (UniqueName: \"kubernetes.io/projected/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-kube-api-access-bs48t\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.711120 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.711091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.711261 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.711132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.711261 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.711158    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs48t\" (UniqueName: \"kubernetes.io/projected/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-kube-api-access-bs48t\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.711261 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.711241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.711434 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.711288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.711434 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.711310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.711603 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.711583    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.711658 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.711615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.711696 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.711669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.713381 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.713361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.713726 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.713709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.727879 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.727853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs48t\" (UniqueName: \"kubernetes.io/projected/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-kube-api-access-bs48t\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:44.847413 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:44.847332    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:45.141773 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:45.141639    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"]
Sep 17 17:13:45.144536 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:13:45.144510    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5acf1e4_7353_4aeb_8cd9_7814f558cd3f.slice/crio-9374abe9a40ba806e3e8ac29a9f4b5244c56683d2d70739497d1711686ef397c WatchSource:0}: Error finding container 9374abe9a40ba806e3e8ac29a9f4b5244c56683d2d70739497d1711686ef397c: Status 404 returned error can't find the container with id 9374abe9a40ba806e3e8ac29a9f4b5244c56683d2d70739497d1711686ef397c
Sep 17 17:13:45.817380 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:45.817346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" event={"ID":"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f","Type":"ContainerStarted","Data":"156ea8d622a4a0b261ad3430e829c217d579cee85a4c2d913ab0733dd0e79a6d"}
Sep 17 17:13:45.817786 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:45.817396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" event={"ID":"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f","Type":"ContainerStarted","Data":"9374abe9a40ba806e3e8ac29a9f4b5244c56683d2d70739497d1711686ef397c"}
Sep 17 17:13:45.848130 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:45.848091    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:54.847577 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:54.847541    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:54.860222 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:54.860197    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:54.871445 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:54.871423    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:13:54.889542 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:13:54.889499    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podStartSLOduration=10.889484236 podStartE2EDuration="10.889484236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:13:45.880372536 +0000 UTC m=+1905.303310399" watchObservedRunningTime="2026-09-17 17:13:54.889484236 +0000 UTC m=+1914.312422122"
Sep 17 17:14:08.111059 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.110469    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"]
Sep 17 17:14:08.121939 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:14:08.121885    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:14:08.121939 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:14:08.121939 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:14:08.121939 ip-10-0-140-49 kubenswrapper[2581]:  > containerID="7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855" cmd=["/bin/sleep","15"]
Sep 17 17:14:08.122232 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.121959    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg" message="" err=<
Sep 17 17:14:08.122232 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:14:08.122232 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:14:08.122232 ip-10-0-140-49 kubenswrapper[2581]:  >
Sep 17 17:14:08.122232 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:14:08.121995    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:14:08.122232 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:14:08.122232 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:14:08.122232 ip-10-0-140-49 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg" podUID="7899f2bb-e5b5-4d0f-bc87-490af33ea684" containerName="main" containerID="cri-o://7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855"
Sep 17 17:14:08.122232 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.122050    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg" podUID="7899f2bb-e5b5-4d0f-bc87-490af33ea684" containerName="main" containerID="cri-o://7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855" gracePeriod=60
Sep 17 17:14:08.465422 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.465395    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:14:08.617037 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.617000    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wtlc\" (UniqueName: \"kubernetes.io/projected/7899f2bb-e5b5-4d0f-bc87-490af33ea684-kube-api-access-8wtlc\") pod \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") "
Sep 17 17:14:08.617223 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.617074    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-model-cache\") pod \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") "
Sep 17 17:14:08.617223 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.617107    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-tmp-dir\") pod \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") "
Sep 17 17:14:08.617223 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.617131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-home\") pod \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") "
Sep 17 17:14:08.617223 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.617161    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7899f2bb-e5b5-4d0f-bc87-490af33ea684-tls-certs\") pod \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") "
Sep 17 17:14:08.617440 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.617238    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-dshm\") pod \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\" (UID: \"7899f2bb-e5b5-4d0f-bc87-490af33ea684\") "
Sep 17 17:14:08.617440 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.617388    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-model-cache" (OuterVolumeSpecName: "model-cache") pod "7899f2bb-e5b5-4d0f-bc87-490af33ea684" (UID: "7899f2bb-e5b5-4d0f-bc87-490af33ea684"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:08.617440 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.617394    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7899f2bb-e5b5-4d0f-bc87-490af33ea684" (UID: "7899f2bb-e5b5-4d0f-bc87-490af33ea684"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:08.617440 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.617413    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-home" (OuterVolumeSpecName: "home") pod "7899f2bb-e5b5-4d0f-bc87-490af33ea684" (UID: "7899f2bb-e5b5-4d0f-bc87-490af33ea684"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:08.617642 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.617566    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:14:08.617642 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.617587    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:14:08.617642 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.617599    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:14:08.619418 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.619391    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7899f2bb-e5b5-4d0f-bc87-490af33ea684-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7899f2bb-e5b5-4d0f-bc87-490af33ea684" (UID: "7899f2bb-e5b5-4d0f-bc87-490af33ea684"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:14:08.619418 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.619404    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-dshm" (OuterVolumeSpecName: "dshm") pod "7899f2bb-e5b5-4d0f-bc87-490af33ea684" (UID: "7899f2bb-e5b5-4d0f-bc87-490af33ea684"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:08.619589 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.619417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7899f2bb-e5b5-4d0f-bc87-490af33ea684-kube-api-access-8wtlc" (OuterVolumeSpecName: "kube-api-access-8wtlc") pod "7899f2bb-e5b5-4d0f-bc87-490af33ea684" (UID: "7899f2bb-e5b5-4d0f-bc87-490af33ea684"). InnerVolumeSpecName "kube-api-access-8wtlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:14:08.718320 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.718275    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7899f2bb-e5b5-4d0f-bc87-490af33ea684-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:14:08.718320 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.718316    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wtlc\" (UniqueName: \"kubernetes.io/projected/7899f2bb-e5b5-4d0f-bc87-490af33ea684-kube-api-access-8wtlc\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:14:08.718320 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.718327    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7899f2bb-e5b5-4d0f-bc87-490af33ea684-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:14:08.927930 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.927894    2581 generic.go:356] "Generic (PLEG): container finished" podID="7899f2bb-e5b5-4d0f-bc87-490af33ea684" containerID="7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855" exitCode=0
Sep 17 17:14:08.928121 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.927946    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"
Sep 17 17:14:08.928121 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.927988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg" event={"ID":"7899f2bb-e5b5-4d0f-bc87-490af33ea684","Type":"ContainerDied","Data":"7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855"}
Sep 17 17:14:08.928121 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.928047    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg" event={"ID":"7899f2bb-e5b5-4d0f-bc87-490af33ea684","Type":"ContainerDied","Data":"8273c252035852792bb3b687c9f091c06e951530df0bf2aeb0562d1891b1ec29"}
Sep 17 17:14:08.928121 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.928068    2581 scope.go:117] "RemoveContainer" containerID="7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855"
Sep 17 17:14:08.937164 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.937145    2581 scope.go:117] "RemoveContainer" containerID="7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855"
Sep 17 17:14:08.937416 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:14:08.937399    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855\": container with ID starting with 7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855 not found: ID does not exist" containerID="7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855"
Sep 17 17:14:08.937474 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.937424    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855"} err="failed to get container status \"7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855\": rpc error: code = NotFound desc = could not find container \"7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855\": container with ID starting with 7730606eba1e4a66bdd6542ecced57acd63b59ed46a34d41ef233e33876d9855 not found: ID does not exist"
Sep 17 17:14:08.957618 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.957593    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"]
Sep 17 17:14:08.965843 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:08.965815    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997kb7xg"]
Sep 17 17:14:10.835682 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:10.835643    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7899f2bb-e5b5-4d0f-bc87-490af33ea684" path="/var/lib/kubelet/pods/7899f2bb-e5b5-4d0f-bc87-490af33ea684/volumes"
Sep 17 17:14:20.086550 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.086515    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"]
Sep 17 17:14:20.087069 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.086925    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7899f2bb-e5b5-4d0f-bc87-490af33ea684" containerName="main"
Sep 17 17:14:20.087069 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.086939    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7899f2bb-e5b5-4d0f-bc87-490af33ea684" containerName="main"
Sep 17 17:14:20.087069 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.087009    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7899f2bb-e5b5-4d0f-bc87-490af33ea684" containerName="main"
Sep 17 17:14:20.091471 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.091450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.094513 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.094356    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 17:14:20.094513 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.094376    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 17 17:14:20.101243 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.101212    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 17:14:20.101368 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.101277    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:14:20.101368 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.101343    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-fb69r\""
Sep 17 17:14:20.101485 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.101399    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 17:14:20.109045 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.109021    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"]
Sep 17 17:14:20.112986 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.112930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.113107 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.112989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eede227-70a7-486f-bfc2-ec7375b5097e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.113107 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.113043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.113107 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.113074    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.113256 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.113155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eede227-70a7-486f-bfc2-ec7375b5097e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.113256 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.113200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.113256 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.113246    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c5vf\" (UniqueName: \"kubernetes.io/projected/8eede227-70a7-486f-bfc2-ec7375b5097e-kube-api-access-2c5vf\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.113393 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.113276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.213814 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.213779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c5vf\" (UniqueName: \"kubernetes.io/projected/8eede227-70a7-486f-bfc2-ec7375b5097e-kube-api-access-2c5vf\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.213992 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.213844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.213992 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.213879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.213992 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.213900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eede227-70a7-486f-bfc2-ec7375b5097e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.213992 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.213944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.214220 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.213993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.214220 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.214078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eede227-70a7-486f-bfc2-ec7375b5097e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.214220 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.214132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.214388 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.214337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.214448 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.214421    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.214448 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.214425    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.214526 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.214475    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.214795 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.214776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eede227-70a7-486f-bfc2-ec7375b5097e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.216294 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.216274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.216436 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.216419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eede227-70a7-486f-bfc2-ec7375b5097e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.223370 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.223345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c5vf\" (UniqueName: \"kubernetes.io/projected/8eede227-70a7-486f-bfc2-ec7375b5097e-kube-api-access-2c5vf\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.330782 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.330746    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"]
Sep 17 17:14:20.336160 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.336134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"
Sep 17 17:14:20.338728 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.338664    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-nxj4l\""
Sep 17 17:14:20.345180 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.345150    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"]
Sep 17 17:14:20.408715 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.408679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:20.416073 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.416044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21a49bf2-c341-41da-9636-0f4ab8ef8775-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4\" (UID: \"21a49bf2-c341-41da-9636-0f4ab8ef8775\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"
Sep 17 17:14:20.416198 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.416086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbcsm\" (UniqueName: \"kubernetes.io/projected/21a49bf2-c341-41da-9636-0f4ab8ef8775-kube-api-access-jbcsm\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4\" (UID: \"21a49bf2-c341-41da-9636-0f4ab8ef8775\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"
Sep 17 17:14:20.516589 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.516558    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21a49bf2-c341-41da-9636-0f4ab8ef8775-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4\" (UID: \"21a49bf2-c341-41da-9636-0f4ab8ef8775\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"
Sep 17 17:14:20.516764 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.516603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbcsm\" (UniqueName: \"kubernetes.io/projected/21a49bf2-c341-41da-9636-0f4ab8ef8775-kube-api-access-jbcsm\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4\" (UID: \"21a49bf2-c341-41da-9636-0f4ab8ef8775\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"
Sep 17 17:14:20.519169 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.519146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21a49bf2-c341-41da-9636-0f4ab8ef8775-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4\" (UID: \"21a49bf2-c341-41da-9636-0f4ab8ef8775\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"
Sep 17 17:14:20.526830 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.526803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbcsm\" (UniqueName: \"kubernetes.io/projected/21a49bf2-c341-41da-9636-0f4ab8ef8775-kube-api-access-jbcsm\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4\" (UID: \"21a49bf2-c341-41da-9636-0f4ab8ef8775\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"
Sep 17 17:14:20.650209 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.650129    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"
Sep 17 17:14:20.693475 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.693443    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"]
Sep 17 17:14:20.696418 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:14:20.696386    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eede227_70a7_486f_bfc2_ec7375b5097e.slice/crio-18608d20539ad60de09f82ceebe6685d2b292725db9c9397074015d5f507ca6f WatchSource:0}: Error finding container 18608d20539ad60de09f82ceebe6685d2b292725db9c9397074015d5f507ca6f: Status 404 returned error can't find the container with id 18608d20539ad60de09f82ceebe6685d2b292725db9c9397074015d5f507ca6f
Sep 17 17:14:20.926220 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.926149    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"]
Sep 17 17:14:20.929499 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:14:20.929474    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a49bf2_c341_41da_9636_0f4ab8ef8775.slice/crio-96883fabcd991cdb46c1e5d4a993f8d0874f0b7894ddbfb366ee8a60038ff643 WatchSource:0}: Error finding container 96883fabcd991cdb46c1e5d4a993f8d0874f0b7894ddbfb366ee8a60038ff643: Status 404 returned error can't find the container with id 96883fabcd991cdb46c1e5d4a993f8d0874f0b7894ddbfb366ee8a60038ff643
Sep 17 17:14:20.983529 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.983492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4" event={"ID":"21a49bf2-c341-41da-9636-0f4ab8ef8775","Type":"ContainerStarted","Data":"96883fabcd991cdb46c1e5d4a993f8d0874f0b7894ddbfb366ee8a60038ff643"}
Sep 17 17:14:20.985087 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.985057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" event={"ID":"8eede227-70a7-486f-bfc2-ec7375b5097e","Type":"ContainerStarted","Data":"66fffd2cac0bde9f66274892ac278ca8c64b4fccd6a9ca1c2aee302449002ec7"}
Sep 17 17:14:20.985193 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:20.985110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" event={"ID":"8eede227-70a7-486f-bfc2-ec7375b5097e","Type":"ContainerStarted","Data":"18608d20539ad60de09f82ceebe6685d2b292725db9c9397074015d5f507ca6f"}
Sep 17 17:14:21.992038 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:21.991993    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4" event={"ID":"21a49bf2-c341-41da-9636-0f4ab8ef8775","Type":"ContainerStarted","Data":"2e50769e257cee04be59adbb545f0f0c5f773e68ad7cc82bb1aa83087b3f521f"}
Sep 17 17:14:21.992461 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:21.992210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"
Sep 17 17:14:22.018659 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:22.018562    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4" podStartSLOduration=2.018522902 podStartE2EDuration="2.018522902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:14:22.014354896 +0000 UTC m=+1941.437292759" watchObservedRunningTime="2026-09-17 17:14:22.018522902 +0000 UTC m=+1941.441460765"
Sep 17 17:14:24.001916 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:24.001882    2581 generic.go:356] "Generic (PLEG): container finished" podID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerID="66fffd2cac0bde9f66274892ac278ca8c64b4fccd6a9ca1c2aee302449002ec7" exitCode=0
Sep 17 17:14:24.002341 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:24.001955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" event={"ID":"8eede227-70a7-486f-bfc2-ec7375b5097e","Type":"ContainerDied","Data":"66fffd2cac0bde9f66274892ac278ca8c64b4fccd6a9ca1c2aee302449002ec7"}
Sep 17 17:14:24.003535 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:24.003520    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:14:41.828164 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:41.828132    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"]
Sep 17 17:14:41.839387 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:14:41.839329    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:14:41.839387 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:14:41.839387 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:14:41.839387 ip-10-0-140-49 kubenswrapper[2581]:  > containerID="156ea8d622a4a0b261ad3430e829c217d579cee85a4c2d913ab0733dd0e79a6d" cmd=["/bin/sleep","15"]
Sep 17 17:14:41.839387 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:41.839387    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" message="" err=<
Sep 17 17:14:41.839387 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:14:41.839387 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:14:41.839387 ip-10-0-140-49 kubenswrapper[2581]:  >
Sep 17 17:14:41.839903 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:14:41.839403    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:14:41.839903 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:14:41.839903 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:14:41.839903 ip-10-0-140-49 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main" containerID="cri-o://156ea8d622a4a0b261ad3430e829c217d579cee85a4c2d913ab0733dd0e79a6d"
Sep 17 17:14:41.839903 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:41.839458    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main" containerID="cri-o://156ea8d622a4a0b261ad3430e829c217d579cee85a4c2d913ab0733dd0e79a6d" gracePeriod=60
Sep 17 17:14:41.861588 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:41.861551    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 17:14:42.860792 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:42.860755    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 17:14:42.861289 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:42.860856    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:14:43.860952 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:43.860857    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 17:14:44.861285 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:44.861244    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 17:14:45.860639 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:45.860596    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 17:14:46.861123 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:46.861077    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 17:14:47.861784 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:47.861738    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 17:14:48.861555 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:48.861508    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 17:14:49.861030 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:49.860961    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 17:14:50.861282 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:50.861241    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Sep 17 17:14:52.166859 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:52.166808    2581 generic.go:356] "Generic (PLEG): container finished" podID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerID="156ea8d622a4a0b261ad3430e829c217d579cee85a4c2d913ab0733dd0e79a6d" exitCode=0
Sep 17 17:14:52.167308 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:52.166878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" event={"ID":"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f","Type":"ContainerDied","Data":"156ea8d622a4a0b261ad3430e829c217d579cee85a4c2d913ab0733dd0e79a6d"}
Sep 17 17:14:52.861740 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:52.861697    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded"
Sep 17 17:14:52.967064 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:52.967041    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:14:52.998116 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:52.998094    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"
Sep 17 17:14:53.115781 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.115714    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-tls-certs\") pod \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") "
Sep 17 17:14:53.115781 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.115759    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-dshm\") pod \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") "
Sep 17 17:14:53.115952 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.115782    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs48t\" (UniqueName: \"kubernetes.io/projected/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-kube-api-access-bs48t\") pod \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") "
Sep 17 17:14:53.115952 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.115808    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-home\") pod \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") "
Sep 17 17:14:53.115952 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.115877    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-tmp-dir\") pod \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") "
Sep 17 17:14:53.115952 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.115933    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-model-cache\") pod \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\" (UID: \"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f\") "
Sep 17 17:14:53.116214 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.116183    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-home" (OuterVolumeSpecName: "home") pod "e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" (UID: "e5acf1e4-7353-4aeb-8cd9-7814f558cd3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:53.116278 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.116209    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" (UID: "e5acf1e4-7353-4aeb-8cd9-7814f558cd3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:53.116278 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.116259    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" (UID: "e5acf1e4-7353-4aeb-8cd9-7814f558cd3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:53.118039 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.118013    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-dshm" (OuterVolumeSpecName: "dshm") pod "e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" (UID: "e5acf1e4-7353-4aeb-8cd9-7814f558cd3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:14:53.118160 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.118068    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-kube-api-access-bs48t" (OuterVolumeSpecName: "kube-api-access-bs48t") pod "e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" (UID: "e5acf1e4-7353-4aeb-8cd9-7814f558cd3f"). InnerVolumeSpecName "kube-api-access-bs48t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:14:53.118160 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.118075    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" (UID: "e5acf1e4-7353-4aeb-8cd9-7814f558cd3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:14:53.172957 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.172935    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"
Sep 17 17:14:53.173349 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.172997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" event={"ID":"e5acf1e4-7353-4aeb-8cd9-7814f558cd3f","Type":"ContainerDied","Data":"9374abe9a40ba806e3e8ac29a9f4b5244c56683d2d70739497d1711686ef397c"}
Sep 17 17:14:53.173349 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.173052    2581 scope.go:117] "RemoveContainer" containerID="156ea8d622a4a0b261ad3430e829c217d579cee85a4c2d913ab0733dd0e79a6d"
Sep 17 17:14:53.195677 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.195652    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"]
Sep 17 17:14:53.200976 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.200949    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr"]
Sep 17 17:14:53.216524 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.216502    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:14:53.216524 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.216522    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:14:53.216656 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.216531    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:14:53.216656 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.216539    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:14:53.216656 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.216547    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:14:53.216656 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.216555    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs48t\" (UniqueName: \"kubernetes.io/projected/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f-kube-api-access-bs48t\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:14:53.862617 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:53.862547    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9956qrfr" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 17:14:54.182784 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:54.182705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" event={"ID":"8eede227-70a7-486f-bfc2-ec7375b5097e","Type":"ContainerStarted","Data":"76bb7f92f73ad4dfcc7b2575792210637dca63906de441ad14d0f89ecbaf3f6d"}
Sep 17 17:14:54.208332 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:54.208275    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" podStartSLOduration=2.605990823 podStartE2EDuration="34.208256478s" totalImagesPullingTime="29.336647657s" totalInitContainerRuntime="2.265617998s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:14:54.206199019 +0000 UTC m=+1973.629136883" watchObservedRunningTime="2026-09-17 17:14:54.208256478 +0000 UTC m=+1973.631194342"
Sep 17 17:14:54.409650 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:54.409615    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:14:54.835679 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:14:54.835646    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" path="/var/lib/kubelet/pods/e5acf1e4-7353-4aeb-8cd9-7814f558cd3f/volumes"
Sep 17 17:15:00.409419 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:15:00.409383    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:15:00.410937 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:15:00.410911    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 17:15:10.409330 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:15:10.409291    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 17:15:20.409059 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:15:20.409013    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 17:15:30.409170 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:15:30.409130    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 17:15:40.409378 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:15:40.409337    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 17:15:50.409040 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:15:50.408997    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 17:16:00.409312 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:16:00.409267    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 17:16:10.409412 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:16:10.409370    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 17:16:20.409382 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:16:20.409290    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 17:16:30.409414 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:16:30.409364    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 17:16:40.419648 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:16:40.419610    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:16:40.427582 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:16:40.427558    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:17:02.884629 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:02.884593    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:17:02.887562 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:02.887537    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:17:09.979779 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:09.979739    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"]
Sep 17 17:17:09.985609 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:09.985581    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"]
Sep 17 17:17:12.998872 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:12.998827    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4" podUID="21a49bf2-c341-41da-9636-0f4ab8ef8775" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:17:22.998407 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:22.998363    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4" podUID="21a49bf2-c341-41da-9636-0f4ab8ef8775" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:17:25.002077 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.002033    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main" containerID="cri-o://76bb7f92f73ad4dfcc7b2575792210637dca63906de441ad14d0f89ecbaf3f6d" gracePeriod=45
Sep 17 17:17:25.003681 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.003649    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4" podUID="21a49bf2-c341-41da-9636-0f4ab8ef8775" containerName="main" containerID="cri-o://2e50769e257cee04be59adbb545f0f0c5f773e68ad7cc82bb1aa83087b3f521f" gracePeriod=45
Sep 17 17:17:25.420606 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.420493    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 17:17:25.661833 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.661807    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:17:25.769659 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.769629    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eede227-70a7-486f-bfc2-ec7375b5097e-tls-certs\") pod \"8eede227-70a7-486f-bfc2-ec7375b5097e\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") "
Sep 17 17:17:25.769659 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.769659    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-kserve-provision-location\") pod \"8eede227-70a7-486f-bfc2-ec7375b5097e\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") "
Sep 17 17:17:25.769900 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.769680    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eede227-70a7-486f-bfc2-ec7375b5097e-cabundle-cert\") pod \"8eede227-70a7-486f-bfc2-ec7375b5097e\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") "
Sep 17 17:17:25.769900 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.769823    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-model-cache\") pod \"8eede227-70a7-486f-bfc2-ec7375b5097e\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") "
Sep 17 17:17:25.769900 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.769852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-home\") pod \"8eede227-70a7-486f-bfc2-ec7375b5097e\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") "
Sep 17 17:17:25.770097 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.769905    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-dshm\") pod \"8eede227-70a7-486f-bfc2-ec7375b5097e\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") "
Sep 17 17:17:25.770097 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.769931    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c5vf\" (UniqueName: \"kubernetes.io/projected/8eede227-70a7-486f-bfc2-ec7375b5097e-kube-api-access-2c5vf\") pod \"8eede227-70a7-486f-bfc2-ec7375b5097e\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") "
Sep 17 17:17:25.770097 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.769988    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-tmp-dir\") pod \"8eede227-70a7-486f-bfc2-ec7375b5097e\" (UID: \"8eede227-70a7-486f-bfc2-ec7375b5097e\") "
Sep 17 17:17:25.770097 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.769994    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eede227-70a7-486f-bfc2-ec7375b5097e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8eede227-70a7-486f-bfc2-ec7375b5097e" (UID: "8eede227-70a7-486f-bfc2-ec7375b5097e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:17:25.770304 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.770157    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-model-cache" (OuterVolumeSpecName: "model-cache") pod "8eede227-70a7-486f-bfc2-ec7375b5097e" (UID: "8eede227-70a7-486f-bfc2-ec7375b5097e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:17:25.770304 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.770268    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:17:25.770304 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.770286    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eede227-70a7-486f-bfc2-ec7375b5097e-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:17:25.770661 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.770628    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-home" (OuterVolumeSpecName: "home") pod "8eede227-70a7-486f-bfc2-ec7375b5097e" (UID: "8eede227-70a7-486f-bfc2-ec7375b5097e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:17:25.772057 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.772029    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eede227-70a7-486f-bfc2-ec7375b5097e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8eede227-70a7-486f-bfc2-ec7375b5097e" (UID: "8eede227-70a7-486f-bfc2-ec7375b5097e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:17:25.772500 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.772476    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-dshm" (OuterVolumeSpecName: "dshm") pod "8eede227-70a7-486f-bfc2-ec7375b5097e" (UID: "8eede227-70a7-486f-bfc2-ec7375b5097e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:17:25.772569 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.772494    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eede227-70a7-486f-bfc2-ec7375b5097e-kube-api-access-2c5vf" (OuterVolumeSpecName: "kube-api-access-2c5vf") pod "8eede227-70a7-486f-bfc2-ec7375b5097e" (UID: "8eede227-70a7-486f-bfc2-ec7375b5097e"). InnerVolumeSpecName "kube-api-access-2c5vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:17:25.781253 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.781225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8eede227-70a7-486f-bfc2-ec7375b5097e" (UID: "8eede227-70a7-486f-bfc2-ec7375b5097e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:17:25.798164 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.798138    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8eede227-70a7-486f-bfc2-ec7375b5097e" (UID: "8eede227-70a7-486f-bfc2-ec7375b5097e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:17:25.871213 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.871187    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eede227-70a7-486f-bfc2-ec7375b5097e-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:17:25.871213 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.871210    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:17:25.871354 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.871218    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:17:25.871354 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.871226    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:17:25.871354 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.871235    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c5vf\" (UniqueName: \"kubernetes.io/projected/8eede227-70a7-486f-bfc2-ec7375b5097e-kube-api-access-2c5vf\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:17:25.871354 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.871243    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eede227-70a7-486f-bfc2-ec7375b5097e-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:17:25.890897 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.890872    2581 generic.go:356] "Generic (PLEG): container finished" podID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerID="76bb7f92f73ad4dfcc7b2575792210637dca63906de441ad14d0f89ecbaf3f6d" exitCode=0
Sep 17 17:17:25.891028 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.890927    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"
Sep 17 17:17:25.891028 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.890937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" event={"ID":"8eede227-70a7-486f-bfc2-ec7375b5097e","Type":"ContainerDied","Data":"76bb7f92f73ad4dfcc7b2575792210637dca63906de441ad14d0f89ecbaf3f6d"}
Sep 17 17:17:25.891028 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.891012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9" event={"ID":"8eede227-70a7-486f-bfc2-ec7375b5097e","Type":"ContainerDied","Data":"18608d20539ad60de09f82ceebe6685d2b292725db9c9397074015d5f507ca6f"}
Sep 17 17:17:25.891202 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.891046    2581 scope.go:117] "RemoveContainer" containerID="76bb7f92f73ad4dfcc7b2575792210637dca63906de441ad14d0f89ecbaf3f6d"
Sep 17 17:17:25.900023 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.900008    2581 scope.go:117] "RemoveContainer" containerID="66fffd2cac0bde9f66274892ac278ca8c64b4fccd6a9ca1c2aee302449002ec7"
Sep 17 17:17:25.907689 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.907673    2581 scope.go:117] "RemoveContainer" containerID="76bb7f92f73ad4dfcc7b2575792210637dca63906de441ad14d0f89ecbaf3f6d"
Sep 17 17:17:25.907928 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:17:25.907913    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76bb7f92f73ad4dfcc7b2575792210637dca63906de441ad14d0f89ecbaf3f6d\": container with ID starting with 76bb7f92f73ad4dfcc7b2575792210637dca63906de441ad14d0f89ecbaf3f6d not found: ID does not exist" containerID="76bb7f92f73ad4dfcc7b2575792210637dca63906de441ad14d0f89ecbaf3f6d"
Sep 17 17:17:25.907994 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.907936    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76bb7f92f73ad4dfcc7b2575792210637dca63906de441ad14d0f89ecbaf3f6d"} err="failed to get container status \"76bb7f92f73ad4dfcc7b2575792210637dca63906de441ad14d0f89ecbaf3f6d\": rpc error: code = NotFound desc = could not find container \"76bb7f92f73ad4dfcc7b2575792210637dca63906de441ad14d0f89ecbaf3f6d\": container with ID starting with 76bb7f92f73ad4dfcc7b2575792210637dca63906de441ad14d0f89ecbaf3f6d not found: ID does not exist"
Sep 17 17:17:25.907994 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.907951    2581 scope.go:117] "RemoveContainer" containerID="66fffd2cac0bde9f66274892ac278ca8c64b4fccd6a9ca1c2aee302449002ec7"
Sep 17 17:17:25.908187 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:17:25.908171    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66fffd2cac0bde9f66274892ac278ca8c64b4fccd6a9ca1c2aee302449002ec7\": container with ID starting with 66fffd2cac0bde9f66274892ac278ca8c64b4fccd6a9ca1c2aee302449002ec7 not found: ID does not exist" containerID="66fffd2cac0bde9f66274892ac278ca8c64b4fccd6a9ca1c2aee302449002ec7"
Sep 17 17:17:25.908232 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.908190    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66fffd2cac0bde9f66274892ac278ca8c64b4fccd6a9ca1c2aee302449002ec7"} err="failed to get container status \"66fffd2cac0bde9f66274892ac278ca8c64b4fccd6a9ca1c2aee302449002ec7\": rpc error: code = NotFound desc = could not find container \"66fffd2cac0bde9f66274892ac278ca8c64b4fccd6a9ca1c2aee302449002ec7\": container with ID starting with 66fffd2cac0bde9f66274892ac278ca8c64b4fccd6a9ca1c2aee302449002ec7 not found: ID does not exist"
Sep 17 17:17:25.920801 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.920773    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"]
Sep 17 17:17:25.924869 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:25.924844    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c6bcf8567tjgk9"]
Sep 17 17:17:26.841999 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:26.841947    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" path="/var/lib/kubelet/pods/8eede227-70a7-486f-bfc2-ec7375b5097e/volumes"
Sep 17 17:17:32.998748 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:32.998693    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4" podUID="21a49bf2-c341-41da-9636-0f4ab8ef8775" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:17:32.999252 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:32.998845    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"
Sep 17 17:17:36.573188 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.573150    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"]
Sep 17 17:17:36.573689 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.573531    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main"
Sep 17 17:17:36.573689 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.573543    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main"
Sep 17 17:17:36.573689 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.573573    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main"
Sep 17 17:17:36.573689 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.573578    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main"
Sep 17 17:17:36.573689 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.573584    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="storage-initializer"
Sep 17 17:17:36.573689 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.573590    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="storage-initializer"
Sep 17 17:17:36.573689 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.573660    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5acf1e4-7353-4aeb-8cd9-7814f558cd3f" containerName="main"
Sep 17 17:17:36.573689 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.573670    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eede227-70a7-486f-bfc2-ec7375b5097e" containerName="main"
Sep 17 17:17:36.575744 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.575721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.579102 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.579076    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 17:17:36.579403 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.579384    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-24xsr\""
Sep 17 17:17:36.579403 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.579390    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 17:17:36.579574 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.579391    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 17:17:36.590248 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.590220    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"]
Sep 17 17:17:36.651175 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.651145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.651280 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.651179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.651280 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.651209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.651280 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.651241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.651387 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.651277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmvc5\" (UniqueName: \"kubernetes.io/projected/c91cf4cd-51da-45d6-bc0a-06b053f80358-kube-api-access-zmvc5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.651387 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.651315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91cf4cd-51da-45d6-bc0a-06b053f80358-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.752164 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.752118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.752273 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.752168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.752273 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.752203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.752273 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.752234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.752273 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.752267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmvc5\" (UniqueName: \"kubernetes.io/projected/c91cf4cd-51da-45d6-bc0a-06b053f80358-kube-api-access-zmvc5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.752476 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.752321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91cf4cd-51da-45d6-bc0a-06b053f80358-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.752623 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.752602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.752667 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.752642    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.752724 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.752688    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.754740 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.754717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.755085 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.755064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91cf4cd-51da-45d6-bc0a-06b053f80358-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.762255 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.762231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmvc5\" (UniqueName: \"kubernetes.io/projected/c91cf4cd-51da-45d6-bc0a-06b053f80358-kube-api-access-zmvc5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:36.886753 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:36.886687    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:37.161628 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:37.161604    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"]
Sep 17 17:17:37.163326 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:17:37.163301    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91cf4cd_51da_45d6_bc0a_06b053f80358.slice/crio-d495537420c89c91392f9e03a225b6fcdebc8332759bb19e3740c0e3bf4aaded WatchSource:0}: Error finding container d495537420c89c91392f9e03a225b6fcdebc8332759bb19e3740c0e3bf4aaded: Status 404 returned error can't find the container with id d495537420c89c91392f9e03a225b6fcdebc8332759bb19e3740c0e3bf4aaded
Sep 17 17:17:37.950760 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:37.950714    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr" event={"ID":"c91cf4cd-51da-45d6-bc0a-06b053f80358","Type":"ContainerStarted","Data":"a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952"}
Sep 17 17:17:37.951232 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:37.950766    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr" event={"ID":"c91cf4cd-51da-45d6-bc0a-06b053f80358","Type":"ContainerStarted","Data":"d495537420c89c91392f9e03a225b6fcdebc8332759bb19e3740c0e3bf4aaded"}
Sep 17 17:17:37.971244 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:37.971199    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr" podStartSLOduration=1.971182893 podStartE2EDuration="1.971182893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:17:37.971139852 +0000 UTC m=+2137.394077715" watchObservedRunningTime="2026-09-17 17:17:37.971182893 +0000 UTC m=+2137.394120759"
Sep 17 17:17:38.887312 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:38.887281    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:42.998238 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:42.998189    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4" podUID="21a49bf2-c341-41da-9636-0f4ab8ef8775" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:17:46.887670 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:46.887627    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:46.900540 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:46.900508    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:47.004903 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:47.004873    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:17:52.998818 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:52.998768    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4" podUID="21a49bf2-c341-41da-9636-0f4ab8ef8775" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:17:55.249341 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:55.249315    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"
Sep 17 17:17:55.400305 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:55.400236    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21a49bf2-c341-41da-9636-0f4ab8ef8775-tls-certs\") pod \"21a49bf2-c341-41da-9636-0f4ab8ef8775\" (UID: \"21a49bf2-c341-41da-9636-0f4ab8ef8775\") "
Sep 17 17:17:55.400305 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:55.400278    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbcsm\" (UniqueName: \"kubernetes.io/projected/21a49bf2-c341-41da-9636-0f4ab8ef8775-kube-api-access-jbcsm\") pod \"21a49bf2-c341-41da-9636-0f4ab8ef8775\" (UID: \"21a49bf2-c341-41da-9636-0f4ab8ef8775\") "
Sep 17 17:17:55.402274 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:55.402247    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a49bf2-c341-41da-9636-0f4ab8ef8775-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21a49bf2-c341-41da-9636-0f4ab8ef8775" (UID: "21a49bf2-c341-41da-9636-0f4ab8ef8775"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:17:55.402405 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:55.402351    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a49bf2-c341-41da-9636-0f4ab8ef8775-kube-api-access-jbcsm" (OuterVolumeSpecName: "kube-api-access-jbcsm") pod "21a49bf2-c341-41da-9636-0f4ab8ef8775" (UID: "21a49bf2-c341-41da-9636-0f4ab8ef8775"). InnerVolumeSpecName "kube-api-access-jbcsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:17:55.501163 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:55.501136    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21a49bf2-c341-41da-9636-0f4ab8ef8775-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:17:55.501163 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:55.501162    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbcsm\" (UniqueName: \"kubernetes.io/projected/21a49bf2-c341-41da-9636-0f4ab8ef8775-kube-api-access-jbcsm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:17:56.047736 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:56.047698    2581 generic.go:356] "Generic (PLEG): container finished" podID="21a49bf2-c341-41da-9636-0f4ab8ef8775" containerID="2e50769e257cee04be59adbb545f0f0c5f773e68ad7cc82bb1aa83087b3f521f" exitCode=0
Sep 17 17:17:56.047884 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:56.047746    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"
Sep 17 17:17:56.047884 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:56.047762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4" event={"ID":"21a49bf2-c341-41da-9636-0f4ab8ef8775","Type":"ContainerDied","Data":"2e50769e257cee04be59adbb545f0f0c5f773e68ad7cc82bb1aa83087b3f521f"}
Sep 17 17:17:56.047884 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:56.047817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4" event={"ID":"21a49bf2-c341-41da-9636-0f4ab8ef8775","Type":"ContainerDied","Data":"96883fabcd991cdb46c1e5d4a993f8d0874f0b7894ddbfb366ee8a60038ff643"}
Sep 17 17:17:56.047884 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:56.047848    2581 scope.go:117] "RemoveContainer" containerID="2e50769e257cee04be59adbb545f0f0c5f773e68ad7cc82bb1aa83087b3f521f"
Sep 17 17:17:56.057872 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:56.057854    2581 scope.go:117] "RemoveContainer" containerID="2e50769e257cee04be59adbb545f0f0c5f773e68ad7cc82bb1aa83087b3f521f"
Sep 17 17:17:56.058209 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:17:56.058187    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e50769e257cee04be59adbb545f0f0c5f773e68ad7cc82bb1aa83087b3f521f\": container with ID starting with 2e50769e257cee04be59adbb545f0f0c5f773e68ad7cc82bb1aa83087b3f521f not found: ID does not exist" containerID="2e50769e257cee04be59adbb545f0f0c5f773e68ad7cc82bb1aa83087b3f521f"
Sep 17 17:17:56.058289 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:56.058223    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e50769e257cee04be59adbb545f0f0c5f773e68ad7cc82bb1aa83087b3f521f"} err="failed to get container status \"2e50769e257cee04be59adbb545f0f0c5f773e68ad7cc82bb1aa83087b3f521f\": rpc error: code = NotFound desc = could not find container \"2e50769e257cee04be59adbb545f0f0c5f773e68ad7cc82bb1aa83087b3f521f\": container with ID starting with 2e50769e257cee04be59adbb545f0f0c5f773e68ad7cc82bb1aa83087b3f521f not found: ID does not exist"
Sep 17 17:17:56.072667 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:56.072642    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"]
Sep 17 17:17:56.077945 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:56.077918    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche7sxs4"]
Sep 17 17:17:56.836866 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:17:56.836831    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21a49bf2-c341-41da-9636-0f4ab8ef8775" path="/var/lib/kubelet/pods/21a49bf2-c341-41da-9636-0f4ab8ef8775/volumes"
Sep 17 17:18:05.683704 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.683668    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"]
Sep 17 17:18:05.684293 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.684272    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a49bf2-c341-41da-9636-0f4ab8ef8775" containerName="main"
Sep 17 17:18:05.684340 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.684297    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a49bf2-c341-41da-9636-0f4ab8ef8775" containerName="main"
Sep 17 17:18:05.684386 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.684376    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a49bf2-c341-41da-9636-0f4ab8ef8775" containerName="main"
Sep 17 17:18:05.687813 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.687791    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.691669 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.691648    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-bd45z\""
Sep 17 17:18:05.691809 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.691791    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 17:18:05.692034 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.692019    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 17:18:05.692092 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.692073    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 17:18:05.700789 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.700767    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"]
Sep 17 17:18:05.784338 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.784309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.784465 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.784351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.784465 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.784376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.784465 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.784414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a85204-e056-4be0-9b26-eaa87e6d47c5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.784465 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.784447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.784682 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.784487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.784682 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.784545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc6jp\" (UniqueName: \"kubernetes.io/projected/48a85204-e056-4be0-9b26-eaa87e6d47c5-kube-api-access-pc6jp\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.885490 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.885466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.885620 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.885499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6jp\" (UniqueName: \"kubernetes.io/projected/48a85204-e056-4be0-9b26-eaa87e6d47c5-kube-api-access-pc6jp\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.885620 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.885531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.885620 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.885556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.885620 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.885579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.885620 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.885601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a85204-e056-4be0-9b26-eaa87e6d47c5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.885878 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.885643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.885937 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.885887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.885937 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.885922    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.886066 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.885998    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.886066 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.886053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.887872 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.887850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.888077 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.888059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a85204-e056-4be0-9b26-eaa87e6d47c5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.896801 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.896775    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6jp\" (UniqueName: \"kubernetes.io/projected/48a85204-e056-4be0-9b26-eaa87e6d47c5-kube-api-access-pc6jp\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:05.998729 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:05.998709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:06.277691 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:06.277593    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"]
Sep 17 17:18:07.103042 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:07.103006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" event={"ID":"48a85204-e056-4be0-9b26-eaa87e6d47c5","Type":"ContainerStarted","Data":"5c14abbcf8cc8efdc3d7c0525c229f9415a6bc21abcb25fd4ba1eb8b2cb4cc99"}
Sep 17 17:18:15.144073 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:15.143992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" event={"ID":"48a85204-e056-4be0-9b26-eaa87e6d47c5","Type":"ContainerStarted","Data":"85bc905f5eeeff469152f248e21e721759d738f3f273b216fd30de91dbc365e0"}
Sep 17 17:18:16.151027 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:16.150919    2581 generic.go:356] "Generic (PLEG): container finished" podID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerID="85bc905f5eeeff469152f248e21e721759d738f3f273b216fd30de91dbc365e0" exitCode=0
Sep 17 17:18:16.151027 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:16.150986    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" event={"ID":"48a85204-e056-4be0-9b26-eaa87e6d47c5","Type":"ContainerDied","Data":"85bc905f5eeeff469152f248e21e721759d738f3f273b216fd30de91dbc365e0"}
Sep 17 17:18:17.159091 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:17.159001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" event={"ID":"48a85204-e056-4be0-9b26-eaa87e6d47c5","Type":"ContainerStarted","Data":"eb12254a77c61d5b5341b4221efc3ec0de12fe723f615f7fc16c79f296b66616"}
Sep 17 17:18:18.165786 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:18.165747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" event={"ID":"48a85204-e056-4be0-9b26-eaa87e6d47c5","Type":"ContainerStarted","Data":"280133fb674e8cecf251643530703684ffb7a6b07cc1077d65587084c4a4b917"}
Sep 17 17:18:18.189533 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:18.189477    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" podStartSLOduration=3.334980395 podStartE2EDuration="13.189458317s" totalImagesPullingTime="8.766326288s" totalInitContainerRuntime="1.088151634s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:18:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:18:18.186998125 +0000 UTC m=+2177.609935989" watchObservedRunningTime="2026-09-17 17:18:18.189458317 +0000 UTC m=+2177.612396182"
Sep 17 17:18:18.999149 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:18.999110    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:25.999120 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:25.999088    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:18:26.000427 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:26.000401    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 17 17:18:33.495024 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.494956    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"]
Sep 17 17:18:33.507241 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:18:33.507165    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:18:33.507241 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:18:33.507241 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:18:33.507241 ip-10-0-140-49 kubenswrapper[2581]:  > containerID="a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952" cmd=["/bin/sleep","15"]
Sep 17 17:18:33.507241 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.507233    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr" message="" err=<
Sep 17 17:18:33.507241 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:18:33.507241 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:18:33.507241 ip-10-0-140-49 kubenswrapper[2581]:  >
Sep 17 17:18:33.507757 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:18:33.507251    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:18:33.507757 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:18:33.507757 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:18:33.507757 ip-10-0-140-49 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr" podUID="c91cf4cd-51da-45d6-bc0a-06b053f80358" containerName="main" containerID="cri-o://a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952"
Sep 17 17:18:33.507757 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.507311    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr" podUID="c91cf4cd-51da-45d6-bc0a-06b053f80358" containerName="main" containerID="cri-o://a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952" gracePeriod=60
Sep 17 17:18:33.858640 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.858610    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:18:33.909424 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.909390    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91cf4cd-51da-45d6-bc0a-06b053f80358-tls-certs\") pod \"c91cf4cd-51da-45d6-bc0a-06b053f80358\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") "
Sep 17 17:18:33.909602 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.909443    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmvc5\" (UniqueName: \"kubernetes.io/projected/c91cf4cd-51da-45d6-bc0a-06b053f80358-kube-api-access-zmvc5\") pod \"c91cf4cd-51da-45d6-bc0a-06b053f80358\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") "
Sep 17 17:18:33.909602 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.909474    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-tmp-dir\") pod \"c91cf4cd-51da-45d6-bc0a-06b053f80358\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") "
Sep 17 17:18:33.909602 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.909578    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-dshm\") pod \"c91cf4cd-51da-45d6-bc0a-06b053f80358\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") "
Sep 17 17:18:33.909767 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.909613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-home\") pod \"c91cf4cd-51da-45d6-bc0a-06b053f80358\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") "
Sep 17 17:18:33.909767 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.909639    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-model-cache\") pod \"c91cf4cd-51da-45d6-bc0a-06b053f80358\" (UID: \"c91cf4cd-51da-45d6-bc0a-06b053f80358\") "
Sep 17 17:18:33.909866 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.909814    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-home" (OuterVolumeSpecName: "home") pod "c91cf4cd-51da-45d6-bc0a-06b053f80358" (UID: "c91cf4cd-51da-45d6-bc0a-06b053f80358"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:18:33.909922 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.909879    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:18:33.910018 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.909995    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-model-cache" (OuterVolumeSpecName: "model-cache") pod "c91cf4cd-51da-45d6-bc0a-06b053f80358" (UID: "c91cf4cd-51da-45d6-bc0a-06b053f80358"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:18:33.910140 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.910114    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c91cf4cd-51da-45d6-bc0a-06b053f80358" (UID: "c91cf4cd-51da-45d6-bc0a-06b053f80358"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:18:33.912430 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.912396    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91cf4cd-51da-45d6-bc0a-06b053f80358-kube-api-access-zmvc5" (OuterVolumeSpecName: "kube-api-access-zmvc5") pod "c91cf4cd-51da-45d6-bc0a-06b053f80358" (UID: "c91cf4cd-51da-45d6-bc0a-06b053f80358"). InnerVolumeSpecName "kube-api-access-zmvc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:18:33.912570 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.912443    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-dshm" (OuterVolumeSpecName: "dshm") pod "c91cf4cd-51da-45d6-bc0a-06b053f80358" (UID: "c91cf4cd-51da-45d6-bc0a-06b053f80358"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:18:33.912570 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:33.912534    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c91cf4cd-51da-45d6-bc0a-06b053f80358-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c91cf4cd-51da-45d6-bc0a-06b053f80358" (UID: "c91cf4cd-51da-45d6-bc0a-06b053f80358"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:18:34.010863 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.010817    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:18:34.010863 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.010849    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:18:34.010863 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.010860    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91cf4cd-51da-45d6-bc0a-06b053f80358-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:18:34.010863 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.010868    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmvc5\" (UniqueName: \"kubernetes.io/projected/c91cf4cd-51da-45d6-bc0a-06b053f80358-kube-api-access-zmvc5\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:18:34.010863 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.010877    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91cf4cd-51da-45d6-bc0a-06b053f80358-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:18:34.242160 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.242124    2581 generic.go:356] "Generic (PLEG): container finished" podID="c91cf4cd-51da-45d6-bc0a-06b053f80358" containerID="a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952" exitCode=0
Sep 17 17:18:34.242378 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.242173    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"
Sep 17 17:18:34.242378 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.242206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr" event={"ID":"c91cf4cd-51da-45d6-bc0a-06b053f80358","Type":"ContainerDied","Data":"a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952"}
Sep 17 17:18:34.242378 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.242262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr" event={"ID":"c91cf4cd-51da-45d6-bc0a-06b053f80358","Type":"ContainerDied","Data":"d495537420c89c91392f9e03a225b6fcdebc8332759bb19e3740c0e3bf4aaded"}
Sep 17 17:18:34.242378 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.242293    2581 scope.go:117] "RemoveContainer" containerID="a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952"
Sep 17 17:18:34.252046 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.252030    2581 scope.go:117] "RemoveContainer" containerID="a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952"
Sep 17 17:18:34.252325 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:18:34.252306    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952\": container with ID starting with a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952 not found: ID does not exist" containerID="a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952"
Sep 17 17:18:34.252383 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.252332    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952"} err="failed to get container status \"a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952\": rpc error: code = NotFound desc = could not find container \"a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952\": container with ID starting with a7b0235eb2b8156118e7fd1ef75589caab8865955da70f0187f48182b5197952 not found: ID does not exist"
Sep 17 17:18:34.266168 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.266143    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"]
Sep 17 17:18:34.271789 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.271769    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-nb2jr"]
Sep 17 17:18:34.835321 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:34.835288    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c91cf4cd-51da-45d6-bc0a-06b053f80358" path="/var/lib/kubelet/pods/c91cf4cd-51da-45d6-bc0a-06b053f80358/volumes"
Sep 17 17:18:35.999328 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:35.999288    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 17 17:18:45.999841 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:45.999798    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 17 17:18:55.999981 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:18:55.999925    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 17 17:19:05.999239 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:19:05.999193    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 17 17:19:16.000161 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:19:16.000081    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 17 17:19:25.999752 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:19:25.999712    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 17 17:19:35.999618 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:19:35.999576    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 17 17:19:46.000162 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:19:46.000114    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 17 17:19:55.999317 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:19:55.999268    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 17 17:20:06.009220 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:06.009190    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:20:06.016858 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:06.016834    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:20:10.396494 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:10.396455    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"]
Sep 17 17:20:10.397027 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:10.396959    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="modelcar" containerID="cri-o://280133fb674e8cecf251643530703684ffb7a6b07cc1077d65587084c4a4b917" gracePeriod=60
Sep 17 17:20:10.706180 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:10.706150    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd_48a85204-e056-4be0-9b26-eaa87e6d47c5/modelcar/0.log"
Sep 17 17:20:10.706807 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:10.706783    2581 generic.go:356] "Generic (PLEG): container finished" podID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerID="280133fb674e8cecf251643530703684ffb7a6b07cc1077d65587084c4a4b917" exitCode=143
Sep 17 17:20:10.706940 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:10.706829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" event={"ID":"48a85204-e056-4be0-9b26-eaa87e6d47c5","Type":"ContainerDied","Data":"280133fb674e8cecf251643530703684ffb7a6b07cc1077d65587084c4a4b917"}
Sep 17 17:20:25.420209 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:25.420147    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main" containerID="cri-o://eb12254a77c61d5b5341b4221efc3ec0de12fe723f615f7fc16c79f296b66616" gracePeriod=45
Sep 17 17:20:26.010400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.010360    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 17 17:20:26.673238 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.673216    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd_48a85204-e056-4be0-9b26-eaa87e6d47c5/modelcar/0.log"
Sep 17 17:20:26.675277 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.675258    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:20:26.720943 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.720913    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-kserve-provision-location\") pod \"48a85204-e056-4be0-9b26-eaa87e6d47c5\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") "
Sep 17 17:20:26.721115 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.720947    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc6jp\" (UniqueName: \"kubernetes.io/projected/48a85204-e056-4be0-9b26-eaa87e6d47c5-kube-api-access-pc6jp\") pod \"48a85204-e056-4be0-9b26-eaa87e6d47c5\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") "
Sep 17 17:20:26.721115 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.721013    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-dshm\") pod \"48a85204-e056-4be0-9b26-eaa87e6d47c5\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") "
Sep 17 17:20:26.721115 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.721036    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-home\") pod \"48a85204-e056-4be0-9b26-eaa87e6d47c5\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") "
Sep 17 17:20:26.721115 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.721091    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-model-cache\") pod \"48a85204-e056-4be0-9b26-eaa87e6d47c5\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") "
Sep 17 17:20:26.721115 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.721113    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a85204-e056-4be0-9b26-eaa87e6d47c5-tls-certs\") pod \"48a85204-e056-4be0-9b26-eaa87e6d47c5\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") "
Sep 17 17:20:26.721368 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.721270    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48a85204-e056-4be0-9b26-eaa87e6d47c5" (UID: "48a85204-e056-4be0-9b26-eaa87e6d47c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:20:26.721472 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.721440    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:20:26.721605 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.721550    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "48a85204-e056-4be0-9b26-eaa87e6d47c5" (UID: "48a85204-e056-4be0-9b26-eaa87e6d47c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:20:26.721833 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.721805    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-home" (OuterVolumeSpecName: "home") pod "48a85204-e056-4be0-9b26-eaa87e6d47c5" (UID: "48a85204-e056-4be0-9b26-eaa87e6d47c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:20:26.723459 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.723429    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a85204-e056-4be0-9b26-eaa87e6d47c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48a85204-e056-4be0-9b26-eaa87e6d47c5" (UID: "48a85204-e056-4be0-9b26-eaa87e6d47c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:20:26.723715 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.723681    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-dshm" (OuterVolumeSpecName: "dshm") pod "48a85204-e056-4be0-9b26-eaa87e6d47c5" (UID: "48a85204-e056-4be0-9b26-eaa87e6d47c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:20:26.723790 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.723770    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a85204-e056-4be0-9b26-eaa87e6d47c5-kube-api-access-pc6jp" (OuterVolumeSpecName: "kube-api-access-pc6jp") pod "48a85204-e056-4be0-9b26-eaa87e6d47c5" (UID: "48a85204-e056-4be0-9b26-eaa87e6d47c5"). InnerVolumeSpecName "kube-api-access-pc6jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:20:26.781866 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.781840    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd_48a85204-e056-4be0-9b26-eaa87e6d47c5/modelcar/0.log"
Sep 17 17:20:26.782494 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.782472    2581 generic.go:356] "Generic (PLEG): container finished" podID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerID="eb12254a77c61d5b5341b4221efc3ec0de12fe723f615f7fc16c79f296b66616" exitCode=0
Sep 17 17:20:26.782557 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.782517    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" event={"ID":"48a85204-e056-4be0-9b26-eaa87e6d47c5","Type":"ContainerDied","Data":"eb12254a77c61d5b5341b4221efc3ec0de12fe723f615f7fc16c79f296b66616"}
Sep 17 17:20:26.782557 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.782538    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"
Sep 17 17:20:26.782636 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.782567    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd" event={"ID":"48a85204-e056-4be0-9b26-eaa87e6d47c5","Type":"ContainerDied","Data":"5c14abbcf8cc8efdc3d7c0525c229f9415a6bc21abcb25fd4ba1eb8b2cb4cc99"}
Sep 17 17:20:26.782636 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.782594    2581 scope.go:117] "RemoveContainer" containerID="280133fb674e8cecf251643530703684ffb7a6b07cc1077d65587084c4a4b917"
Sep 17 17:20:26.791077 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.791060    2581 scope.go:117] "RemoveContainer" containerID="eb12254a77c61d5b5341b4221efc3ec0de12fe723f615f7fc16c79f296b66616"
Sep 17 17:20:26.798900 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.798882    2581 scope.go:117] "RemoveContainer" containerID="85bc905f5eeeff469152f248e21e721759d738f3f273b216fd30de91dbc365e0"
Sep 17 17:20:26.806170 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.806151    2581 scope.go:117] "RemoveContainer" containerID="280133fb674e8cecf251643530703684ffb7a6b07cc1077d65587084c4a4b917"
Sep 17 17:20:26.806475 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:20:26.806443    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280133fb674e8cecf251643530703684ffb7a6b07cc1077d65587084c4a4b917\": container with ID starting with 280133fb674e8cecf251643530703684ffb7a6b07cc1077d65587084c4a4b917 not found: ID does not exist" containerID="280133fb674e8cecf251643530703684ffb7a6b07cc1077d65587084c4a4b917"
Sep 17 17:20:26.806563 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.806479    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280133fb674e8cecf251643530703684ffb7a6b07cc1077d65587084c4a4b917"} err="failed to get container status \"280133fb674e8cecf251643530703684ffb7a6b07cc1077d65587084c4a4b917\": rpc error: code = NotFound desc = could not find container \"280133fb674e8cecf251643530703684ffb7a6b07cc1077d65587084c4a4b917\": container with ID starting with 280133fb674e8cecf251643530703684ffb7a6b07cc1077d65587084c4a4b917 not found: ID does not exist"
Sep 17 17:20:26.806563 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.806498    2581 scope.go:117] "RemoveContainer" containerID="eb12254a77c61d5b5341b4221efc3ec0de12fe723f615f7fc16c79f296b66616"
Sep 17 17:20:26.806740 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:20:26.806725    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb12254a77c61d5b5341b4221efc3ec0de12fe723f615f7fc16c79f296b66616\": container with ID starting with eb12254a77c61d5b5341b4221efc3ec0de12fe723f615f7fc16c79f296b66616 not found: ID does not exist" containerID="eb12254a77c61d5b5341b4221efc3ec0de12fe723f615f7fc16c79f296b66616"
Sep 17 17:20:26.806780 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.806743    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb12254a77c61d5b5341b4221efc3ec0de12fe723f615f7fc16c79f296b66616"} err="failed to get container status \"eb12254a77c61d5b5341b4221efc3ec0de12fe723f615f7fc16c79f296b66616\": rpc error: code = NotFound desc = could not find container \"eb12254a77c61d5b5341b4221efc3ec0de12fe723f615f7fc16c79f296b66616\": container with ID starting with eb12254a77c61d5b5341b4221efc3ec0de12fe723f615f7fc16c79f296b66616 not found: ID does not exist"
Sep 17 17:20:26.806780 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.806756    2581 scope.go:117] "RemoveContainer" containerID="85bc905f5eeeff469152f248e21e721759d738f3f273b216fd30de91dbc365e0"
Sep 17 17:20:26.806952 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:20:26.806939    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85bc905f5eeeff469152f248e21e721759d738f3f273b216fd30de91dbc365e0\": container with ID starting with 85bc905f5eeeff469152f248e21e721759d738f3f273b216fd30de91dbc365e0 not found: ID does not exist" containerID="85bc905f5eeeff469152f248e21e721759d738f3f273b216fd30de91dbc365e0"
Sep 17 17:20:26.807023 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.806956    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85bc905f5eeeff469152f248e21e721759d738f3f273b216fd30de91dbc365e0"} err="failed to get container status \"85bc905f5eeeff469152f248e21e721759d738f3f273b216fd30de91dbc365e0\": rpc error: code = NotFound desc = could not find container \"85bc905f5eeeff469152f248e21e721759d738f3f273b216fd30de91dbc365e0\": container with ID starting with 85bc905f5eeeff469152f248e21e721759d738f3f273b216fd30de91dbc365e0 not found: ID does not exist"
Sep 17 17:20:26.822340 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.822317    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-tmp-dir\") pod \"48a85204-e056-4be0-9b26-eaa87e6d47c5\" (UID: \"48a85204-e056-4be0-9b26-eaa87e6d47c5\") "
Sep 17 17:20:26.822471 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.822460    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:20:26.822521 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.822472    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:20:26.822521 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.822480    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:20:26.822521 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.822490    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a85204-e056-4be0-9b26-eaa87e6d47c5-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:20:26.822521 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.822497    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc6jp\" (UniqueName: \"kubernetes.io/projected/48a85204-e056-4be0-9b26-eaa87e6d47c5-kube-api-access-pc6jp\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:20:26.833471 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.833443    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48a85204-e056-4be0-9b26-eaa87e6d47c5" (UID: "48a85204-e056-4be0-9b26-eaa87e6d47c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:20:26.923143 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:26.923109    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a85204-e056-4be0-9b26-eaa87e6d47c5-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:20:27.105152 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:27.105073    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"]
Sep 17 17:20:27.116214 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:27.116186    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d75bmd"]
Sep 17 17:20:28.835141 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:20:28.835104    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" path="/var/lib/kubelet/pods/48a85204-e056-4be0-9b26-eaa87e6d47c5/volumes"
Sep 17 17:21:08.061159 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.061126    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"]
Sep 17 17:21:08.061563 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.061452    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="modelcar-init"
Sep 17 17:21:08.061563 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.061463    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="modelcar-init"
Sep 17 17:21:08.061563 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.061477    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main"
Sep 17 17:21:08.061563 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.061483    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main"
Sep 17 17:21:08.061563 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.061496    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c91cf4cd-51da-45d6-bc0a-06b053f80358" containerName="main"
Sep 17 17:21:08.061563 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.061501    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c91cf4cd-51da-45d6-bc0a-06b053f80358" containerName="main"
Sep 17 17:21:08.061563 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.061508    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="modelcar"
Sep 17 17:21:08.061563 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.061513    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="modelcar"
Sep 17 17:21:08.061563 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.061562    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="main"
Sep 17 17:21:08.061563 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.061569    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a85204-e056-4be0-9b26-eaa87e6d47c5" containerName="modelcar"
Sep 17 17:21:08.061878 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.061577    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c91cf4cd-51da-45d6-bc0a-06b053f80358" containerName="main"
Sep 17 17:21:08.065788 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.065768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.070330 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.070312    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 17:21:08.075508 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.075484    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:21:08.075752 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.075731    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-bss4l\""
Sep 17 17:21:08.075817 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.075750    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 17:21:08.075817 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.075731    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 17 17:21:08.076521 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.076502    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 17:21:08.097272 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.097240    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"]
Sep 17 17:21:08.184349 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.184311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-dshm\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.184349 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.184355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-cabundle-cert\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.184561 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.184413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-home\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.184561 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.184445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.184561 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.184497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-tls-certs\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.184561 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.184529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-model-cache\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.184561 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.184547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrsnr\" (UniqueName: \"kubernetes.io/projected/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-kube-api-access-vrsnr\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.184731 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.184567    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-tmp-dir\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.285345 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.285303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-home\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.285550 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.285355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.285550 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.285376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-tls-certs\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.285550 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.285400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-model-cache\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.285550 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.285427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrsnr\" (UniqueName: \"kubernetes.io/projected/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-kube-api-access-vrsnr\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.285550 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.285451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-tmp-dir\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.285550 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.285516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-dshm\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.285881 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.285553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-cabundle-cert\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.285881 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.285801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-home\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.285881 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.285833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.286043 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.285884    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-model-cache\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.286043 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.285955    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-tmp-dir\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.286326 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.286305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-cabundle-cert\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.287905 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.287885    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-dshm\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.288232 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.288213    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-tls-certs\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.299857 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.299826    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrsnr\" (UniqueName: \"kubernetes.io/projected/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-kube-api-access-vrsnr\") pod \"custom-route-timeout-test-kserve-654bb97cf4-5whwd\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.375597 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.375518    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:08.382812 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.382784    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"]
Sep 17 17:21:08.386351 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.386331    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"
Sep 17 17:21:08.389841 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.389815    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-5mds9\""
Sep 17 17:21:08.397191 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.397162    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"]
Sep 17 17:21:08.487620 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.487579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4hhv\" (UniqueName: \"kubernetes.io/projected/7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106-kube-api-access-d4hhv\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j\" (UID: \"7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"
Sep 17 17:21:08.487809 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.487632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j\" (UID: \"7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"
Sep 17 17:21:08.588669 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.588631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4hhv\" (UniqueName: \"kubernetes.io/projected/7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106-kube-api-access-d4hhv\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j\" (UID: \"7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"
Sep 17 17:21:08.588669 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.588679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j\" (UID: \"7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"
Sep 17 17:21:08.591195 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.591167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j\" (UID: \"7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"
Sep 17 17:21:08.604749 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.604728    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4hhv\" (UniqueName: \"kubernetes.io/projected/7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106-kube-api-access-d4hhv\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j\" (UID: \"7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"
Sep 17 17:21:08.665605 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.665578    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"]
Sep 17 17:21:08.667061 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:21:08.667036    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe74c25_061e_41cc_8c00_c9aa3d5e8729.slice/crio-6119c858083fe1e62f01a660cab4775def91651609440b77a42aeadf2901e08f WatchSource:0}: Error finding container 6119c858083fe1e62f01a660cab4775def91651609440b77a42aeadf2901e08f: Status 404 returned error can't find the container with id 6119c858083fe1e62f01a660cab4775def91651609440b77a42aeadf2901e08f
Sep 17 17:21:08.710864 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.710838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"
Sep 17 17:21:08.967673 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.967634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" event={"ID":"4fe74c25-061e-41cc-8c00-c9aa3d5e8729","Type":"ContainerStarted","Data":"fd639af22ba7c7167a75662969020c3e4060354204acff3c22db561ec111726b"}
Sep 17 17:21:08.967866 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.967689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" event={"ID":"4fe74c25-061e-41cc-8c00-c9aa3d5e8729","Type":"ContainerStarted","Data":"6119c858083fe1e62f01a660cab4775def91651609440b77a42aeadf2901e08f"}
Sep 17 17:21:08.995291 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:08.995255    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"]
Sep 17 17:21:08.996480 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:21:08.996448    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbb7a66_7d6f_46a7_a596_1b3fa0dcd106.slice/crio-194c24606a5123d6ce80c708407b83af77f892366fd1ee294b3a184252b8f9dd WatchSource:0}: Error finding container 194c24606a5123d6ce80c708407b83af77f892366fd1ee294b3a184252b8f9dd: Status 404 returned error can't find the container with id 194c24606a5123d6ce80c708407b83af77f892366fd1ee294b3a184252b8f9dd
Sep 17 17:21:09.973761 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:09.973729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j" event={"ID":"7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106","Type":"ContainerStarted","Data":"6a9e65ba25aec0a5a1e0ff10d09866b773d5ea5b9cee257e99c64b4b3f3851e2"}
Sep 17 17:21:09.974230 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:09.973780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j" event={"ID":"7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106","Type":"ContainerStarted","Data":"194c24606a5123d6ce80c708407b83af77f892366fd1ee294b3a184252b8f9dd"}
Sep 17 17:21:09.974230 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:09.974091    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"
Sep 17 17:21:09.995066 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:09.995022    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j" podStartSLOduration=1.995006097 podStartE2EDuration="1.995006097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:21:09.99314614 +0000 UTC m=+2349.416084015" watchObservedRunningTime="2026-09-17 17:21:09.995006097 +0000 UTC m=+2349.417943965"
Sep 17 17:21:10.979239 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:10.979207    2581 generic.go:356] "Generic (PLEG): container finished" podID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerID="fd639af22ba7c7167a75662969020c3e4060354204acff3c22db561ec111726b" exitCode=0
Sep 17 17:21:10.979634 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:10.979280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" event={"ID":"4fe74c25-061e-41cc-8c00-c9aa3d5e8729","Type":"ContainerDied","Data":"fd639af22ba7c7167a75662969020c3e4060354204acff3c22db561ec111726b"}
Sep 17 17:21:11.986336 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:11.986301    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" event={"ID":"4fe74c25-061e-41cc-8c00-c9aa3d5e8729","Type":"ContainerStarted","Data":"043b2c424fa70f5d3ab486af73295bf70c22e13e1f834a67b3318fa208dfeb65"}
Sep 17 17:21:12.014840 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:12.014776    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" podStartSLOduration=1.79583816 podStartE2EDuration="4.01475386s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.2189157s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:21:12.013264301 +0000 UTC m=+2351.436202179" watchObservedRunningTime="2026-09-17 17:21:12.01475386 +0000 UTC m=+2351.437691737"
Sep 17 17:21:12.376159 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:12.376075    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:18.376778 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:18.376743    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:21:18.378363 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:18.378329    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 17:21:28.376379 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:28.376332    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 17:21:38.376345 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:38.376291    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 17:21:40.981580 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:40.981551    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"
Sep 17 17:21:48.376596 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:48.376555    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 17:21:58.376397 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:21:58.376353    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 17:22:02.913633 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:22:02.913605    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:22:02.917075 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:22:02.917052    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:22:08.376919 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:22:08.376873    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 17:22:18.376802 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:22:18.376705    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 17:22:28.376204 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:22:28.376164    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 17:22:38.376926 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:22:38.376881    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 17:22:48.376250 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:22:48.376204    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 17:22:58.387177 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:22:58.387142    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:22:58.396800 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:22:58.396761    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:23:13.722540 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:13.722506    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"]
Sep 17 17:23:13.728543 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:13.728512    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"]
Sep 17 17:23:20.981316 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:20.981274    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j" podUID="7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:23:28.744404 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:28.744336    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j" podUID="7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106" containerName="main" containerID="cri-o://6a9e65ba25aec0a5a1e0ff10d09866b773d5ea5b9cee257e99c64b4b3f3851e2" gracePeriod=45
Sep 17 17:23:28.752040 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:28.752006    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main" containerID="cri-o://043b2c424fa70f5d3ab486af73295bf70c22e13e1f834a67b3318fa208dfeb65" gracePeriod=45
Sep 17 17:23:29.406169 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.406147    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:23:29.448871 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.448847    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-kserve-provision-location\") pod \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") "
Sep 17 17:23:29.448999 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.448903    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-model-cache\") pod \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") "
Sep 17 17:23:29.448999 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.448930    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-dshm\") pod \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") "
Sep 17 17:23:29.448999 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.448946    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-tls-certs\") pod \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") "
Sep 17 17:23:29.448999 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.448996    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-tmp-dir\") pod \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") "
Sep 17 17:23:29.449214 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.449029    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-cabundle-cert\") pod \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") "
Sep 17 17:23:29.449214 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.449114    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-home\") pod \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") "
Sep 17 17:23:29.449214 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.449177    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrsnr\" (UniqueName: \"kubernetes.io/projected/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-kube-api-access-vrsnr\") pod \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\" (UID: \"4fe74c25-061e-41cc-8c00-c9aa3d5e8729\") "
Sep 17 17:23:29.449378 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.449327    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fe74c25-061e-41cc-8c00-c9aa3d5e8729" (UID: "4fe74c25-061e-41cc-8c00-c9aa3d5e8729"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:23:29.449439 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.449399    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4fe74c25-061e-41cc-8c00-c9aa3d5e8729" (UID: "4fe74c25-061e-41cc-8c00-c9aa3d5e8729"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:23:29.449510 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.449486    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:23:29.449510 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.449507    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:23:29.450143 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.450115    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-home" (OuterVolumeSpecName: "home") pod "4fe74c25-061e-41cc-8c00-c9aa3d5e8729" (UID: "4fe74c25-061e-41cc-8c00-c9aa3d5e8729"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:23:29.452123 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.452059    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fe74c25-061e-41cc-8c00-c9aa3d5e8729" (UID: "4fe74c25-061e-41cc-8c00-c9aa3d5e8729"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:23:29.452290 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.452261    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-dshm" (OuterVolumeSpecName: "dshm") pod "4fe74c25-061e-41cc-8c00-c9aa3d5e8729" (UID: "4fe74c25-061e-41cc-8c00-c9aa3d5e8729"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:23:29.452430 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.452357    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-kube-api-access-vrsnr" (OuterVolumeSpecName: "kube-api-access-vrsnr") pod "4fe74c25-061e-41cc-8c00-c9aa3d5e8729" (UID: "4fe74c25-061e-41cc-8c00-c9aa3d5e8729"). InnerVolumeSpecName "kube-api-access-vrsnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:23:29.466784 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.466758    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fe74c25-061e-41cc-8c00-c9aa3d5e8729" (UID: "4fe74c25-061e-41cc-8c00-c9aa3d5e8729"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:23:29.476495 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.476471    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4fe74c25-061e-41cc-8c00-c9aa3d5e8729" (UID: "4fe74c25-061e-41cc-8c00-c9aa3d5e8729"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:23:29.550627 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.550574    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:23:29.550627 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.550594    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:23:29.550627 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.550609    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:23:29.550627 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.550616    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:23:29.550627 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.550625    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrsnr\" (UniqueName: \"kubernetes.io/projected/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-kube-api-access-vrsnr\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:23:29.550849 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.550633    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fe74c25-061e-41cc-8c00-c9aa3d5e8729-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:23:29.652469 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.652442    2581 generic.go:356] "Generic (PLEG): container finished" podID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerID="043b2c424fa70f5d3ab486af73295bf70c22e13e1f834a67b3318fa208dfeb65" exitCode=0
Sep 17 17:23:29.652591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.652506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" event={"ID":"4fe74c25-061e-41cc-8c00-c9aa3d5e8729","Type":"ContainerDied","Data":"043b2c424fa70f5d3ab486af73295bf70c22e13e1f834a67b3318fa208dfeb65"}
Sep 17 17:23:29.652591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.652514    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"
Sep 17 17:23:29.652591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.652570    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" event={"ID":"4fe74c25-061e-41cc-8c00-c9aa3d5e8729","Type":"ContainerDied","Data":"6119c858083fe1e62f01a660cab4775def91651609440b77a42aeadf2901e08f"}
Sep 17 17:23:29.652591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.652592    2581 scope.go:117] "RemoveContainer" containerID="043b2c424fa70f5d3ab486af73295bf70c22e13e1f834a67b3318fa208dfeb65"
Sep 17 17:23:29.662298 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.662281    2581 scope.go:117] "RemoveContainer" containerID="fd639af22ba7c7167a75662969020c3e4060354204acff3c22db561ec111726b"
Sep 17 17:23:29.670055 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.670035    2581 scope.go:117] "RemoveContainer" containerID="043b2c424fa70f5d3ab486af73295bf70c22e13e1f834a67b3318fa208dfeb65"
Sep 17 17:23:29.670313 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:23:29.670294    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043b2c424fa70f5d3ab486af73295bf70c22e13e1f834a67b3318fa208dfeb65\": container with ID starting with 043b2c424fa70f5d3ab486af73295bf70c22e13e1f834a67b3318fa208dfeb65 not found: ID does not exist" containerID="043b2c424fa70f5d3ab486af73295bf70c22e13e1f834a67b3318fa208dfeb65"
Sep 17 17:23:29.670367 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.670321    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043b2c424fa70f5d3ab486af73295bf70c22e13e1f834a67b3318fa208dfeb65"} err="failed to get container status \"043b2c424fa70f5d3ab486af73295bf70c22e13e1f834a67b3318fa208dfeb65\": rpc error: code = NotFound desc = could not find container \"043b2c424fa70f5d3ab486af73295bf70c22e13e1f834a67b3318fa208dfeb65\": container with ID starting with 043b2c424fa70f5d3ab486af73295bf70c22e13e1f834a67b3318fa208dfeb65 not found: ID does not exist"
Sep 17 17:23:29.670367 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.670338    2581 scope.go:117] "RemoveContainer" containerID="fd639af22ba7c7167a75662969020c3e4060354204acff3c22db561ec111726b"
Sep 17 17:23:29.670588 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:23:29.670573    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd639af22ba7c7167a75662969020c3e4060354204acff3c22db561ec111726b\": container with ID starting with fd639af22ba7c7167a75662969020c3e4060354204acff3c22db561ec111726b not found: ID does not exist" containerID="fd639af22ba7c7167a75662969020c3e4060354204acff3c22db561ec111726b"
Sep 17 17:23:29.670656 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.670590    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd639af22ba7c7167a75662969020c3e4060354204acff3c22db561ec111726b"} err="failed to get container status \"fd639af22ba7c7167a75662969020c3e4060354204acff3c22db561ec111726b\": rpc error: code = NotFound desc = could not find container \"fd639af22ba7c7167a75662969020c3e4060354204acff3c22db561ec111726b\": container with ID starting with fd639af22ba7c7167a75662969020c3e4060354204acff3c22db561ec111726b not found: ID does not exist"
Sep 17 17:23:29.683230 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.683206    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"]
Sep 17 17:23:29.692079 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:29.692060    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd"]
Sep 17 17:23:30.387851 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:30.387816    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-654bb97cf4-5whwd" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 17:23:30.836075 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:30.836038    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" path="/var/lib/kubelet/pods/4fe74c25-061e-41cc-8c00-c9aa3d5e8729/volumes"
Sep 17 17:23:30.981217 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:30.981187    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j" podUID="7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:23:40.981728 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:40.981674    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j" podUID="7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:23:40.982200 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:40.981795    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"
Sep 17 17:23:50.981902 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:50.981807    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j" podUID="7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:23:58.987410 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:58.987367    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"
Sep 17 17:23:59.071146 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.071086    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4hhv\" (UniqueName: \"kubernetes.io/projected/7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106-kube-api-access-d4hhv\") pod \"7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106\" (UID: \"7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106\") "
Sep 17 17:23:59.071146 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.071117    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106-tls-certs\") pod \"7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106\" (UID: \"7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106\") "
Sep 17 17:23:59.073013 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.072980    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106" (UID: "7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:23:59.073127 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.073049    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106-kube-api-access-d4hhv" (OuterVolumeSpecName: "kube-api-access-d4hhv") pod "7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106" (UID: "7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106"). InnerVolumeSpecName "kube-api-access-d4hhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:23:59.172630 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.172606    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4hhv\" (UniqueName: \"kubernetes.io/projected/7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106-kube-api-access-d4hhv\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:23:59.172630 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.172626    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:23:59.804468 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.804428    2581 generic.go:356] "Generic (PLEG): container finished" podID="7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106" containerID="6a9e65ba25aec0a5a1e0ff10d09866b773d5ea5b9cee257e99c64b4b3f3851e2" exitCode=0
Sep 17 17:23:59.804654 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.804505    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"
Sep 17 17:23:59.804654 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.804514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j" event={"ID":"7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106","Type":"ContainerDied","Data":"6a9e65ba25aec0a5a1e0ff10d09866b773d5ea5b9cee257e99c64b4b3f3851e2"}
Sep 17 17:23:59.804654 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.804599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j" event={"ID":"7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106","Type":"ContainerDied","Data":"194c24606a5123d6ce80c708407b83af77f892366fd1ee294b3a184252b8f9dd"}
Sep 17 17:23:59.804654 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.804624    2581 scope.go:117] "RemoveContainer" containerID="6a9e65ba25aec0a5a1e0ff10d09866b773d5ea5b9cee257e99c64b4b3f3851e2"
Sep 17 17:23:59.815189 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.815169    2581 scope.go:117] "RemoveContainer" containerID="6a9e65ba25aec0a5a1e0ff10d09866b773d5ea5b9cee257e99c64b4b3f3851e2"
Sep 17 17:23:59.815491 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:23:59.815470    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9e65ba25aec0a5a1e0ff10d09866b773d5ea5b9cee257e99c64b4b3f3851e2\": container with ID starting with 6a9e65ba25aec0a5a1e0ff10d09866b773d5ea5b9cee257e99c64b4b3f3851e2 not found: ID does not exist" containerID="6a9e65ba25aec0a5a1e0ff10d09866b773d5ea5b9cee257e99c64b4b3f3851e2"
Sep 17 17:23:59.815542 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.815500    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9e65ba25aec0a5a1e0ff10d09866b773d5ea5b9cee257e99c64b4b3f3851e2"} err="failed to get container status \"6a9e65ba25aec0a5a1e0ff10d09866b773d5ea5b9cee257e99c64b4b3f3851e2\": rpc error: code = NotFound desc = could not find container \"6a9e65ba25aec0a5a1e0ff10d09866b773d5ea5b9cee257e99c64b4b3f3851e2\": container with ID starting with 6a9e65ba25aec0a5a1e0ff10d09866b773d5ea5b9cee257e99c64b4b3f3851e2 not found: ID does not exist"
Sep 17 17:23:59.837975 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.837945    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"]
Sep 17 17:23:59.847242 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:23:59.847215    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58wgx6j"]
Sep 17 17:24:00.834906 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:00.834876    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106" path="/var/lib/kubelet/pods/7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106/volumes"
Sep 17 17:24:25.139994 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.139944    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"]
Sep 17 17:24:25.140429 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.140293    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106" containerName="main"
Sep 17 17:24:25.140429 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.140305    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106" containerName="main"
Sep 17 17:24:25.140429 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.140319    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="storage-initializer"
Sep 17 17:24:25.140429 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.140325    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="storage-initializer"
Sep 17 17:24:25.140429 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.140335    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main"
Sep 17 17:24:25.140429 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.140341    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main"
Sep 17 17:24:25.140429 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.140394    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cbb7a66-7d6f-46a7-a596-1b3fa0dcd106" containerName="main"
Sep 17 17:24:25.140429 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.140400    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe74c25-061e-41cc-8c00-c9aa3d5e8729" containerName="main"
Sep 17 17:24:25.144944 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.144922    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.156805 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.156785    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 17:24:25.157227 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.157210    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:24:25.158072 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.158058    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 17 17:24:25.158385 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.158365    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 17:24:25.158528 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.158502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-tls-certs\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.158617 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.158579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7prd\" (UniqueName: \"kubernetes.io/projected/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-kube-api-access-g7prd\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.158696 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.158676    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-home\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.158696 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.158684    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-54gwq\""
Sep 17 17:24:25.158858 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.158701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-model-cache\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.158858 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.158708    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 17:24:25.158858 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.158739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-tmp-dir\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.158858 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.158835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-cabundle-cert\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.159051 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.158870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-dshm\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.159051 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.158909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-kserve-provision-location\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.187349 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.187324    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"]
Sep 17 17:24:25.259423 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.259395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-home\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.259517 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.259424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-model-cache\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.259517 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.259454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-tmp-dir\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.259517 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.259483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-cabundle-cert\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.259517 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.259501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-dshm\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.259714 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.259522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-kserve-provision-location\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.259714 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.259677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-tls-certs\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.259826 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.259716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7prd\" (UniqueName: \"kubernetes.io/projected/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-kube-api-access-g7prd\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.259826 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.259809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-home\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.259937 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.259830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-model-cache\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.259937 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.259910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-kserve-provision-location\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.260095 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.260068    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-tmp-dir\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.260313 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.260296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-cabundle-cert\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.261881 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.261860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-dshm\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.262249 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.262229    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-tls-certs\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.273185 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.273164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7prd\" (UniqueName: \"kubernetes.io/projected/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-kube-api-access-g7prd\") pod \"router-with-refs-test-kserve-cc757d4b-55sbb\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.454793 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.454769    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:25.660061 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.660032    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"]
Sep 17 17:24:25.664533 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.664510    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"
Sep 17 17:24:25.666937 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.666914    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-mmsnh\""
Sep 17 17:24:25.684598 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.684571    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"]
Sep 17 17:24:25.745618 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.745543    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"]
Sep 17 17:24:25.748268 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:24:25.748246    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760f9479_5eb6_4eb9_a2b2_1ebc2769a502.slice/crio-6c74bce30cba57225391f4d2482eaf5e1aa3651496ca7e65c5dfd02916599fea WatchSource:0}: Error finding container 6c74bce30cba57225391f4d2482eaf5e1aa3651496ca7e65c5dfd02916599fea: Status 404 returned error can't find the container with id 6c74bce30cba57225391f4d2482eaf5e1aa3651496ca7e65c5dfd02916599fea
Sep 17 17:24:25.763887 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.763861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f85b5d38-101b-4542-b771-d69702dda767-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7\" (UID: \"f85b5d38-101b-4542-b771-d69702dda767\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"
Sep 17 17:24:25.764015 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.763906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7cc7\" (UniqueName: \"kubernetes.io/projected/f85b5d38-101b-4542-b771-d69702dda767-kube-api-access-p7cc7\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7\" (UID: \"f85b5d38-101b-4542-b771-d69702dda767\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"
Sep 17 17:24:25.865103 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.865077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f85b5d38-101b-4542-b771-d69702dda767-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7\" (UID: \"f85b5d38-101b-4542-b771-d69702dda767\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"
Sep 17 17:24:25.865236 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.865115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7cc7\" (UniqueName: \"kubernetes.io/projected/f85b5d38-101b-4542-b771-d69702dda767-kube-api-access-p7cc7\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7\" (UID: \"f85b5d38-101b-4542-b771-d69702dda767\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"
Sep 17 17:24:25.867559 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.867522    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f85b5d38-101b-4542-b771-d69702dda767-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7\" (UID: \"f85b5d38-101b-4542-b771-d69702dda767\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"
Sep 17 17:24:25.875095 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.875071    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7cc7\" (UniqueName: \"kubernetes.io/projected/f85b5d38-101b-4542-b771-d69702dda767-kube-api-access-p7cc7\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7\" (UID: \"f85b5d38-101b-4542-b771-d69702dda767\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"
Sep 17 17:24:25.935375 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.935345    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" event={"ID":"760f9479-5eb6-4eb9-a2b2-1ebc2769a502","Type":"ContainerStarted","Data":"bf407da2075fd301cdaeb6c9da583c2a7cd6b05961c6e1314274d9281ef9a523"}
Sep 17 17:24:25.935488 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.935403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" event={"ID":"760f9479-5eb6-4eb9-a2b2-1ebc2769a502","Type":"ContainerStarted","Data":"6c74bce30cba57225391f4d2482eaf5e1aa3651496ca7e65c5dfd02916599fea"}
Sep 17 17:24:25.975289 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:25.975258    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"
Sep 17 17:24:26.259142 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:26.259119    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"]
Sep 17 17:24:26.260925 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:24:26.260898    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85b5d38_101b_4542_b771_d69702dda767.slice/crio-1f53428aaee4130192d01a4fb1e76f5e7184b1284f219cc4c7a8f0df3990a34e WatchSource:0}: Error finding container 1f53428aaee4130192d01a4fb1e76f5e7184b1284f219cc4c7a8f0df3990a34e: Status 404 returned error can't find the container with id 1f53428aaee4130192d01a4fb1e76f5e7184b1284f219cc4c7a8f0df3990a34e
Sep 17 17:24:26.940879 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:26.940838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7" event={"ID":"f85b5d38-101b-4542-b771-d69702dda767","Type":"ContainerStarted","Data":"64af1063befbad5f8df60b252ea54b44ec3d8392614c82e095f030e44fb830d1"}
Sep 17 17:24:26.941087 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:26.940901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7" event={"ID":"f85b5d38-101b-4542-b771-d69702dda767","Type":"ContainerStarted","Data":"1f53428aaee4130192d01a4fb1e76f5e7184b1284f219cc4c7a8f0df3990a34e"}
Sep 17 17:24:26.941163 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:26.941095    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"
Sep 17 17:24:28.951865 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:28.951827    2581 generic.go:356] "Generic (PLEG): container finished" podID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerID="bf407da2075fd301cdaeb6c9da583c2a7cd6b05961c6e1314274d9281ef9a523" exitCode=0
Sep 17 17:24:28.952289 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:28.951899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" event={"ID":"760f9479-5eb6-4eb9-a2b2-1ebc2769a502","Type":"ContainerDied","Data":"bf407da2075fd301cdaeb6c9da583c2a7cd6b05961c6e1314274d9281ef9a523"}
Sep 17 17:24:28.986818 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:28.986766    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7" podStartSLOduration=3.986746954 podStartE2EDuration="3.986746954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:24:26.968777783 +0000 UTC m=+2546.391715647" watchObservedRunningTime="2026-09-17 17:24:28.986746954 +0000 UTC m=+2548.409684818"
Sep 17 17:24:29.958935 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:29.958900    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" event={"ID":"760f9479-5eb6-4eb9-a2b2-1ebc2769a502","Type":"ContainerStarted","Data":"6af9c1a9761198b99a116dc1c4bf7f4acc34a7595e285fae8680b4c7cd390561"}
Sep 17 17:24:29.985618 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:29.985542    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" podStartSLOduration=2.82554217 podStartE2EDuration="4.985523815s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.159981645s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:24:29.983876623 +0000 UTC m=+2549.406814489" watchObservedRunningTime="2026-09-17 17:24:29.985523815 +0000 UTC m=+2549.408461677"
Sep 17 17:24:30.454893 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:30.454861    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:35.455470 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:35.455426    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:24:35.457181 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:35.457153    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 17:24:45.455180 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:45.455124    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 17:24:55.455605 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:55.455563    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 17:24:57.947513 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:24:57.947484    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"
Sep 17 17:25:05.455391 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:25:05.455349    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 17:25:15.455767 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:25:15.455676    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 17:25:25.455718 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:25:25.455672    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 17:25:35.455418 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:25:35.455364    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 17:25:45.455336 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:25:45.455282    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 17:25:55.455843 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:25:55.455801    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 17:26:05.455768 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:05.455710    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 17 17:26:15.465843 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:15.465803    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:26:15.473868 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:15.473841    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:26:20.670272 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:20.670242    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"]
Sep 17 17:26:20.672051 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:20.672027    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"]
Sep 17 17:26:27.947402 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:27.947355    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7" podUID="f85b5d38-101b-4542-b771-d69702dda767" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:26:35.690934 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:35.690878    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7" podUID="f85b5d38-101b-4542-b771-d69702dda767" containerName="main" containerID="cri-o://64af1063befbad5f8df60b252ea54b44ec3d8392614c82e095f030e44fb830d1" gracePeriod=45
Sep 17 17:26:35.692573 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:35.692522    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="main" containerID="cri-o://6af9c1a9761198b99a116dc1c4bf7f4acc34a7595e285fae8680b4c7cd390561" gracePeriod=45
Sep 17 17:26:36.339865 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.339843    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:26:36.514028 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.514004    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-model-cache\") pod \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") "
Sep 17 17:26:36.514150 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.514046    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-dshm\") pod \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") "
Sep 17 17:26:36.514150 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.514084    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-kserve-provision-location\") pod \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") "
Sep 17 17:26:36.514150 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.514100    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-home\") pod \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") "
Sep 17 17:26:36.514150 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.514115    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-tmp-dir\") pod \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") "
Sep 17 17:26:36.514376 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.514197    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-cabundle-cert\") pod \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") "
Sep 17 17:26:36.514376 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.514256    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-tls-certs\") pod \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") "
Sep 17 17:26:36.514376 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.514288    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7prd\" (UniqueName: \"kubernetes.io/projected/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-kube-api-access-g7prd\") pod \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\" (UID: \"760f9479-5eb6-4eb9-a2b2-1ebc2769a502\") "
Sep 17 17:26:36.514537 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.514365    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-model-cache" (OuterVolumeSpecName: "model-cache") pod "760f9479-5eb6-4eb9-a2b2-1ebc2769a502" (UID: "760f9479-5eb6-4eb9-a2b2-1ebc2769a502"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:26:36.514537 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.514532    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:26:36.514662 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.514591    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "760f9479-5eb6-4eb9-a2b2-1ebc2769a502" (UID: "760f9479-5eb6-4eb9-a2b2-1ebc2769a502"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:26:36.515082 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.515051    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-home" (OuterVolumeSpecName: "home") pod "760f9479-5eb6-4eb9-a2b2-1ebc2769a502" (UID: "760f9479-5eb6-4eb9-a2b2-1ebc2769a502"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:26:36.516617 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.516588    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-kube-api-access-g7prd" (OuterVolumeSpecName: "kube-api-access-g7prd") pod "760f9479-5eb6-4eb9-a2b2-1ebc2769a502" (UID: "760f9479-5eb6-4eb9-a2b2-1ebc2769a502"). InnerVolumeSpecName "kube-api-access-g7prd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:26:36.516617 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.516597    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "760f9479-5eb6-4eb9-a2b2-1ebc2769a502" (UID: "760f9479-5eb6-4eb9-a2b2-1ebc2769a502"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:26:36.516765 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.516730    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-dshm" (OuterVolumeSpecName: "dshm") pod "760f9479-5eb6-4eb9-a2b2-1ebc2769a502" (UID: "760f9479-5eb6-4eb9-a2b2-1ebc2769a502"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:26:36.526400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.526370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "760f9479-5eb6-4eb9-a2b2-1ebc2769a502" (UID: "760f9479-5eb6-4eb9-a2b2-1ebc2769a502"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:26:36.537489 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.537469    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "760f9479-5eb6-4eb9-a2b2-1ebc2769a502" (UID: "760f9479-5eb6-4eb9-a2b2-1ebc2769a502"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:26:36.566357 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.566328    2581 generic.go:356] "Generic (PLEG): container finished" podID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerID="6af9c1a9761198b99a116dc1c4bf7f4acc34a7595e285fae8680b4c7cd390561" exitCode=0
Sep 17 17:26:36.566473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.566388    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"
Sep 17 17:26:36.566473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.566394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" event={"ID":"760f9479-5eb6-4eb9-a2b2-1ebc2769a502","Type":"ContainerDied","Data":"6af9c1a9761198b99a116dc1c4bf7f4acc34a7595e285fae8680b4c7cd390561"}
Sep 17 17:26:36.566473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.566437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb" event={"ID":"760f9479-5eb6-4eb9-a2b2-1ebc2769a502","Type":"ContainerDied","Data":"6c74bce30cba57225391f4d2482eaf5e1aa3651496ca7e65c5dfd02916599fea"}
Sep 17 17:26:36.566473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.566458    2581 scope.go:117] "RemoveContainer" containerID="6af9c1a9761198b99a116dc1c4bf7f4acc34a7595e285fae8680b4c7cd390561"
Sep 17 17:26:36.575025 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.575008    2581 scope.go:117] "RemoveContainer" containerID="bf407da2075fd301cdaeb6c9da583c2a7cd6b05961c6e1314274d9281ef9a523"
Sep 17 17:26:36.582240 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.582224    2581 scope.go:117] "RemoveContainer" containerID="6af9c1a9761198b99a116dc1c4bf7f4acc34a7595e285fae8680b4c7cd390561"
Sep 17 17:26:36.582491 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:26:36.582475    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af9c1a9761198b99a116dc1c4bf7f4acc34a7595e285fae8680b4c7cd390561\": container with ID starting with 6af9c1a9761198b99a116dc1c4bf7f4acc34a7595e285fae8680b4c7cd390561 not found: ID does not exist" containerID="6af9c1a9761198b99a116dc1c4bf7f4acc34a7595e285fae8680b4c7cd390561"
Sep 17 17:26:36.582539 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.582498    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af9c1a9761198b99a116dc1c4bf7f4acc34a7595e285fae8680b4c7cd390561"} err="failed to get container status \"6af9c1a9761198b99a116dc1c4bf7f4acc34a7595e285fae8680b4c7cd390561\": rpc error: code = NotFound desc = could not find container \"6af9c1a9761198b99a116dc1c4bf7f4acc34a7595e285fae8680b4c7cd390561\": container with ID starting with 6af9c1a9761198b99a116dc1c4bf7f4acc34a7595e285fae8680b4c7cd390561 not found: ID does not exist"
Sep 17 17:26:36.582539 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.582513    2581 scope.go:117] "RemoveContainer" containerID="bf407da2075fd301cdaeb6c9da583c2a7cd6b05961c6e1314274d9281ef9a523"
Sep 17 17:26:36.582754 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:26:36.582733    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf407da2075fd301cdaeb6c9da583c2a7cd6b05961c6e1314274d9281ef9a523\": container with ID starting with bf407da2075fd301cdaeb6c9da583c2a7cd6b05961c6e1314274d9281ef9a523 not found: ID does not exist" containerID="bf407da2075fd301cdaeb6c9da583c2a7cd6b05961c6e1314274d9281ef9a523"
Sep 17 17:26:36.582816 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.582764    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf407da2075fd301cdaeb6c9da583c2a7cd6b05961c6e1314274d9281ef9a523"} err="failed to get container status \"bf407da2075fd301cdaeb6c9da583c2a7cd6b05961c6e1314274d9281ef9a523\": rpc error: code = NotFound desc = could not find container \"bf407da2075fd301cdaeb6c9da583c2a7cd6b05961c6e1314274d9281ef9a523\": container with ID starting with bf407da2075fd301cdaeb6c9da583c2a7cd6b05961c6e1314274d9281ef9a523 not found: ID does not exist"
Sep 17 17:26:36.593983 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.593948    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"]
Sep 17 17:26:36.605907 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.605885    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cc757d4b-55sbb"]
Sep 17 17:26:36.615407 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.615389    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:26:36.615475 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.615409    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:26:36.615475 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.615420    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:26:36.615475 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.615428    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:26:36.615475 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.615437    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:26:36.615475 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.615445    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7prd\" (UniqueName: \"kubernetes.io/projected/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-kube-api-access-g7prd\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:26:36.615475 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.615458    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/760f9479-5eb6-4eb9-a2b2-1ebc2769a502-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:26:36.835517 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:36.835464    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" path="/var/lib/kubelet/pods/760f9479-5eb6-4eb9-a2b2-1ebc2769a502/volumes"
Sep 17 17:26:37.947455 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:37.947411    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7" podUID="f85b5d38-101b-4542-b771-d69702dda767" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:26:47.947428 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:47.947324    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7" podUID="f85b5d38-101b-4542-b771-d69702dda767" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:26:47.947837 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:47.947473    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"
Sep 17 17:26:57.947788 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:26:57.947736    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7" podUID="f85b5d38-101b-4542-b771-d69702dda767" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:27:02.941470 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:02.941440    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:27:02.949134 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:02.949111    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:27:05.935331 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:05.935301    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"
Sep 17 17:27:06.030522 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.030462    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7cc7\" (UniqueName: \"kubernetes.io/projected/f85b5d38-101b-4542-b771-d69702dda767-kube-api-access-p7cc7\") pod \"f85b5d38-101b-4542-b771-d69702dda767\" (UID: \"f85b5d38-101b-4542-b771-d69702dda767\") "
Sep 17 17:27:06.030631 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.030545    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f85b5d38-101b-4542-b771-d69702dda767-tls-certs\") pod \"f85b5d38-101b-4542-b771-d69702dda767\" (UID: \"f85b5d38-101b-4542-b771-d69702dda767\") "
Sep 17 17:27:06.032449 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.032413    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85b5d38-101b-4542-b771-d69702dda767-kube-api-access-p7cc7" (OuterVolumeSpecName: "kube-api-access-p7cc7") pod "f85b5d38-101b-4542-b771-d69702dda767" (UID: "f85b5d38-101b-4542-b771-d69702dda767"). InnerVolumeSpecName "kube-api-access-p7cc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:27:06.032609 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.032470    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f85b5d38-101b-4542-b771-d69702dda767-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f85b5d38-101b-4542-b771-d69702dda767" (UID: "f85b5d38-101b-4542-b771-d69702dda767"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:27:06.131571 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.131549    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f85b5d38-101b-4542-b771-d69702dda767-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:27:06.131571 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.131569    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7cc7\" (UniqueName: \"kubernetes.io/projected/f85b5d38-101b-4542-b771-d69702dda767-kube-api-access-p7cc7\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:27:06.707744 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.707710    2581 generic.go:356] "Generic (PLEG): container finished" podID="f85b5d38-101b-4542-b771-d69702dda767" containerID="64af1063befbad5f8df60b252ea54b44ec3d8392614c82e095f030e44fb830d1" exitCode=0
Sep 17 17:27:06.707893 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.707766    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"
Sep 17 17:27:06.707893 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.707810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7" event={"ID":"f85b5d38-101b-4542-b771-d69702dda767","Type":"ContainerDied","Data":"64af1063befbad5f8df60b252ea54b44ec3d8392614c82e095f030e44fb830d1"}
Sep 17 17:27:06.707893 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.707873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7" event={"ID":"f85b5d38-101b-4542-b771-d69702dda767","Type":"ContainerDied","Data":"1f53428aaee4130192d01a4fb1e76f5e7184b1284f219cc4c7a8f0df3990a34e"}
Sep 17 17:27:06.708040 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.707901    2581 scope.go:117] "RemoveContainer" containerID="64af1063befbad5f8df60b252ea54b44ec3d8392614c82e095f030e44fb830d1"
Sep 17 17:27:06.718716 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.718697    2581 scope.go:117] "RemoveContainer" containerID="64af1063befbad5f8df60b252ea54b44ec3d8392614c82e095f030e44fb830d1"
Sep 17 17:27:06.719019 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:27:06.718999    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64af1063befbad5f8df60b252ea54b44ec3d8392614c82e095f030e44fb830d1\": container with ID starting with 64af1063befbad5f8df60b252ea54b44ec3d8392614c82e095f030e44fb830d1 not found: ID does not exist" containerID="64af1063befbad5f8df60b252ea54b44ec3d8392614c82e095f030e44fb830d1"
Sep 17 17:27:06.719110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.719031    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64af1063befbad5f8df60b252ea54b44ec3d8392614c82e095f030e44fb830d1"} err="failed to get container status \"64af1063befbad5f8df60b252ea54b44ec3d8392614c82e095f030e44fb830d1\": rpc error: code = NotFound desc = could not find container \"64af1063befbad5f8df60b252ea54b44ec3d8392614c82e095f030e44fb830d1\": container with ID starting with 64af1063befbad5f8df60b252ea54b44ec3d8392614c82e095f030e44fb830d1 not found: ID does not exist"
Sep 17 17:27:06.752840 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.752812    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"]
Sep 17 17:27:06.755913 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.755887    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-k7pk7"]
Sep 17 17:27:06.834800 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:06.834778    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f85b5d38-101b-4542-b771-d69702dda767" path="/var/lib/kubelet/pods/f85b5d38-101b-4542-b771-d69702dda767/volumes"
Sep 17 17:27:15.924099 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.924066    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"]
Sep 17 17:27:15.924534 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.924510    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f85b5d38-101b-4542-b771-d69702dda767" containerName="main"
Sep 17 17:27:15.924534 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.924530    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f85b5d38-101b-4542-b771-d69702dda767" containerName="main"
Sep 17 17:27:15.924608 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.924552    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="storage-initializer"
Sep 17 17:27:15.924608 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.924560    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="storage-initializer"
Sep 17 17:27:15.924608 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.924585    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="main"
Sep 17 17:27:15.924608 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.924594    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="main"
Sep 17 17:27:15.924738 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.924696    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f85b5d38-101b-4542-b771-d69702dda767" containerName="main"
Sep 17 17:27:15.924738 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.924706    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="760f9479-5eb6-4eb9-a2b2-1ebc2769a502" containerName="main"
Sep 17 17:27:15.928090 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.928064    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:15.931612 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.931589    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 17:27:15.931744 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.931647    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 17:27:15.932803 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.932787    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:27:15.937360 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.937323    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 17:27:15.937464 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.937361    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-r6hb9\""
Sep 17 17:27:15.937464 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.937433    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 17:27:15.937634 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.937621    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-54r96\""
Sep 17 17:27:15.962223 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:15.962187    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"]
Sep 17 17:27:16.005457 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.005416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/836459a1-5af1-4cfc-a831-90815433a201-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.005457 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.005460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.005708 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.005501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.005708 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.005528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8zfk\" (UniqueName: \"kubernetes.io/projected/836459a1-5af1-4cfc-a831-90815433a201-kube-api-access-c8zfk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.005708 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.005550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.005708 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.005588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/836459a1-5af1-4cfc-a831-90815433a201-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.005708 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.005681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.005894 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.005734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.107069 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.107026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.107069 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.107068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8zfk\" (UniqueName: \"kubernetes.io/projected/836459a1-5af1-4cfc-a831-90815433a201-kube-api-access-c8zfk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.107332 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.107103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.107332 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.107130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/836459a1-5af1-4cfc-a831-90815433a201-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.107332 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.107166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.107332 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.107239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.107332 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.107273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/836459a1-5af1-4cfc-a831-90815433a201-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.107332 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.107309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.107654 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.107583    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.107654 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.107626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.107775 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.107669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.107854 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.107812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.107906 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.107887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/836459a1-5af1-4cfc-a831-90815433a201-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.109502 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.109483    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.109892 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.109869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/836459a1-5af1-4cfc-a831-90815433a201-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.119838 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.119811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8zfk\" (UniqueName: \"kubernetes.io/projected/836459a1-5af1-4cfc-a831-90815433a201-kube-api-access-c8zfk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.237727 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.237685    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.325537 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.325505    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"]
Sep 17 17:27:16.328665 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.328646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"
Sep 17 17:27:16.334198 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.334173    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-p6v7b\""
Sep 17 17:27:16.347591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.347568    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"]
Sep 17 17:27:16.409547 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.409507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1e9f131-eba3-4309-94d2-54cf90b0f7e4-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8\" (UID: \"d1e9f131-eba3-4309-94d2-54cf90b0f7e4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"
Sep 17 17:27:16.409547 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.409551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n4k4\" (UniqueName: \"kubernetes.io/projected/d1e9f131-eba3-4309-94d2-54cf90b0f7e4-kube-api-access-5n4k4\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8\" (UID: \"d1e9f131-eba3-4309-94d2-54cf90b0f7e4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"
Sep 17 17:27:16.510492 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.510405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1e9f131-eba3-4309-94d2-54cf90b0f7e4-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8\" (UID: \"d1e9f131-eba3-4309-94d2-54cf90b0f7e4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"
Sep 17 17:27:16.510492 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.510453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n4k4\" (UniqueName: \"kubernetes.io/projected/d1e9f131-eba3-4309-94d2-54cf90b0f7e4-kube-api-access-5n4k4\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8\" (UID: \"d1e9f131-eba3-4309-94d2-54cf90b0f7e4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"
Sep 17 17:27:16.512886 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.512855    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1e9f131-eba3-4309-94d2-54cf90b0f7e4-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8\" (UID: \"d1e9f131-eba3-4309-94d2-54cf90b0f7e4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"
Sep 17 17:27:16.527472 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.527427    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"]
Sep 17 17:27:16.529308 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:27:16.529279    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836459a1_5af1_4cfc_a831_90815433a201.slice/crio-f8f6b16a9c200ced26cb9880217da788ded564ec23166ab088648f5e8bc0819d WatchSource:0}: Error finding container f8f6b16a9c200ced26cb9880217da788ded564ec23166ab088648f5e8bc0819d: Status 404 returned error can't find the container with id f8f6b16a9c200ced26cb9880217da788ded564ec23166ab088648f5e8bc0819d
Sep 17 17:27:16.531452 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.531425    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n4k4\" (UniqueName: \"kubernetes.io/projected/d1e9f131-eba3-4309-94d2-54cf90b0f7e4-kube-api-access-5n4k4\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8\" (UID: \"d1e9f131-eba3-4309-94d2-54cf90b0f7e4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"
Sep 17 17:27:16.640847 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.640811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"
Sep 17 17:27:16.759277 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.759240    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" event={"ID":"836459a1-5af1-4cfc-a831-90815433a201","Type":"ContainerStarted","Data":"4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996"}
Sep 17 17:27:16.759473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.759294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" event={"ID":"836459a1-5af1-4cfc-a831-90815433a201","Type":"ContainerStarted","Data":"f8f6b16a9c200ced26cb9880217da788ded564ec23166ab088648f5e8bc0819d"}
Sep 17 17:27:16.759473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.759327    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:16.921786 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:16.921753    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"]
Sep 17 17:27:16.926182 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:27:16.926154    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e9f131_eba3_4309_94d2_54cf90b0f7e4.slice/crio-5d273f3a1a6a4ffc454a8c9cd0ae1bfbebbc9999b6bd40b22c0d5da22f765bfd WatchSource:0}: Error finding container 5d273f3a1a6a4ffc454a8c9cd0ae1bfbebbc9999b6bd40b22c0d5da22f765bfd: Status 404 returned error can't find the container with id 5d273f3a1a6a4ffc454a8c9cd0ae1bfbebbc9999b6bd40b22c0d5da22f765bfd
Sep 17 17:27:17.765582 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:17.765533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8" event={"ID":"d1e9f131-eba3-4309-94d2-54cf90b0f7e4","Type":"ContainerStarted","Data":"4f1ca31da2f244cb94d00c9c36d1e3e798d63df6c27a3e291e1ae58e80d22b56"}
Sep 17 17:27:17.765798 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:17.765606    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"
Sep 17 17:27:17.765798 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:17.765630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8" event={"ID":"d1e9f131-eba3-4309-94d2-54cf90b0f7e4","Type":"ContainerStarted","Data":"5d273f3a1a6a4ffc454a8c9cd0ae1bfbebbc9999b6bd40b22c0d5da22f765bfd"}
Sep 17 17:27:17.767369 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:17.767344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" event={"ID":"836459a1-5af1-4cfc-a831-90815433a201","Type":"ContainerStarted","Data":"957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396"}
Sep 17 17:27:17.800470 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:17.800421    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8" podStartSLOduration=1.800407126 podStartE2EDuration="1.800407126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:27:17.798635751 +0000 UTC m=+2717.221573595" watchObservedRunningTime="2026-09-17 17:27:17.800407126 +0000 UTC m=+2717.223344989"
Sep 17 17:27:19.778490 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:19.778445    2581 generic.go:356] "Generic (PLEG): container finished" podID="836459a1-5af1-4cfc-a831-90815433a201" containerID="957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396" exitCode=0
Sep 17 17:27:19.778894 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:19.778516    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" event={"ID":"836459a1-5af1-4cfc-a831-90815433a201","Type":"ContainerDied","Data":"957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396"}
Sep 17 17:27:20.785790 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:20.785751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" event={"ID":"836459a1-5af1-4cfc-a831-90815433a201","Type":"ContainerStarted","Data":"71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3"}
Sep 17 17:27:20.817534 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:20.817311    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" podStartSLOduration=3.554365117 podStartE2EDuration="5.8172941s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.262928983s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:27:20.815785495 +0000 UTC m=+2720.238723358" watchObservedRunningTime="2026-09-17 17:27:20.8172941 +0000 UTC m=+2720.240231963"
Sep 17 17:27:26.237919 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:26.237877    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:26.238428 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:26.237947    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:26.239400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:26.239358    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8001/health\": dial tcp 10.134.0.56:8001: connect: connection refused"
Sep 17 17:27:26.251287 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:26.251253    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:27:36.238266 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:36.238207    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8001/health\": dial tcp 10.134.0.56:8001: connect: connection refused"
Sep 17 17:27:46.238425 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:46.238368    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8001/health\": dial tcp 10.134.0.56:8001: connect: connection refused"
Sep 17 17:27:48.773735 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:48.773703    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"
Sep 17 17:27:56.238650 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:27:56.238594    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8001/health\": dial tcp 10.134.0.56:8001: connect: connection refused"
Sep 17 17:28:06.238400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:28:06.238350    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8001/health\": dial tcp 10.134.0.56:8001: connect: connection refused"
Sep 17 17:28:16.238837 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:28:16.238475    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8001/health\": dial tcp 10.134.0.56:8001: connect: connection refused"
Sep 17 17:28:26.238273 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:28:26.238230    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8001/health\": dial tcp 10.134.0.56:8001: connect: connection refused"
Sep 17 17:28:36.238522 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:28:36.238477    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8001/health\": dial tcp 10.134.0.56:8001: connect: connection refused"
Sep 17 17:28:46.238830 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:28:46.238767    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8001/health\": dial tcp 10.134.0.56:8001: connect: connection refused"
Sep 17 17:28:56.238282 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:28:56.238233    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8001/health\": dial tcp 10.134.0.56:8001: connect: connection refused"
Sep 17 17:29:06.238567 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:06.238517    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8001/health\": dial tcp 10.134.0.56:8001: connect: connection refused"
Sep 17 17:29:16.248685 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:16.248646    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:29:16.260674 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:16.260648    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:29:36.117175 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:36.117138    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"]
Sep 17 17:29:36.118834 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:36.118802    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"]
Sep 17 17:29:38.774068 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:38.774023    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8" podUID="d1e9f131-eba3-4309-94d2-54cf90b0f7e4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:29:48.774253 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:48.774152    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8" podUID="d1e9f131-eba3-4309-94d2-54cf90b0f7e4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:29:51.137093 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:51.137049    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8" podUID="d1e9f131-eba3-4309-94d2-54cf90b0f7e4" containerName="main" containerID="cri-o://4f1ca31da2f244cb94d00c9c36d1e3e798d63df6c27a3e291e1ae58e80d22b56" gracePeriod=45
Sep 17 17:29:51.139203 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:51.139156    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main" containerID="cri-o://71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3" gracePeriod=45
Sep 17 17:29:51.755892 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:51.755834    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="llm-d-routing-sidecar" containerID="cri-o://4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996" gracePeriod=45
Sep 17 17:29:51.996267 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:51.996237    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:29:52.046314 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.046289    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-dshm\") pod \"836459a1-5af1-4cfc-a831-90815433a201\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") "
Sep 17 17:29:52.046464 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.046320    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/836459a1-5af1-4cfc-a831-90815433a201-tls-certs\") pod \"836459a1-5af1-4cfc-a831-90815433a201\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") "
Sep 17 17:29:52.046464 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.046335    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-tmp-dir\") pod \"836459a1-5af1-4cfc-a831-90815433a201\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") "
Sep 17 17:29:52.046589 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.046458    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-kserve-provision-location\") pod \"836459a1-5af1-4cfc-a831-90815433a201\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") "
Sep 17 17:29:52.046589 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.046507    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8zfk\" (UniqueName: \"kubernetes.io/projected/836459a1-5af1-4cfc-a831-90815433a201-kube-api-access-c8zfk\") pod \"836459a1-5af1-4cfc-a831-90815433a201\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") "
Sep 17 17:29:52.046589 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.046534    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-model-cache\") pod \"836459a1-5af1-4cfc-a831-90815433a201\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") "
Sep 17 17:29:52.046769 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.046595    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-home\") pod \"836459a1-5af1-4cfc-a831-90815433a201\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") "
Sep 17 17:29:52.046769 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.046646    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/836459a1-5af1-4cfc-a831-90815433a201-cabundle-cert\") pod \"836459a1-5af1-4cfc-a831-90815433a201\" (UID: \"836459a1-5af1-4cfc-a831-90815433a201\") "
Sep 17 17:29:52.047438 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.047384    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836459a1-5af1-4cfc-a831-90815433a201-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "836459a1-5af1-4cfc-a831-90815433a201" (UID: "836459a1-5af1-4cfc-a831-90815433a201"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:29:52.047740 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.047709    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-home" (OuterVolumeSpecName: "home") pod "836459a1-5af1-4cfc-a831-90815433a201" (UID: "836459a1-5af1-4cfc-a831-90815433a201"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:29:52.047839 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.047800    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-model-cache" (OuterVolumeSpecName: "model-cache") pod "836459a1-5af1-4cfc-a831-90815433a201" (UID: "836459a1-5af1-4cfc-a831-90815433a201"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:29:52.048832 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.048784    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836459a1-5af1-4cfc-a831-90815433a201-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "836459a1-5af1-4cfc-a831-90815433a201" (UID: "836459a1-5af1-4cfc-a831-90815433a201"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:29:52.048992 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.048891    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-dshm" (OuterVolumeSpecName: "dshm") pod "836459a1-5af1-4cfc-a831-90815433a201" (UID: "836459a1-5af1-4cfc-a831-90815433a201"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:29:52.048992 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.048897    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836459a1-5af1-4cfc-a831-90815433a201-kube-api-access-c8zfk" (OuterVolumeSpecName: "kube-api-access-c8zfk") pod "836459a1-5af1-4cfc-a831-90815433a201" (UID: "836459a1-5af1-4cfc-a831-90815433a201"). InnerVolumeSpecName "kube-api-access-c8zfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:29:52.063848 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.063819    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "836459a1-5af1-4cfc-a831-90815433a201" (UID: "836459a1-5af1-4cfc-a831-90815433a201"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:29:52.074825 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.074801    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "836459a1-5af1-4cfc-a831-90815433a201" (UID: "836459a1-5af1-4cfc-a831-90815433a201"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:29:52.148005 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.147983    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/836459a1-5af1-4cfc-a831-90815433a201-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.148005 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.148003    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.148322 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.148011    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/836459a1-5af1-4cfc-a831-90815433a201-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.148322 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.148019    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.148322 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.148027    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.148322 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.148034    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8zfk\" (UniqueName: \"kubernetes.io/projected/836459a1-5af1-4cfc-a831-90815433a201-kube-api-access-c8zfk\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.148322 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.148042    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.148322 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.148050    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/836459a1-5af1-4cfc-a831-90815433a201-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:29:52.518879 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.518843    2581 generic.go:356] "Generic (PLEG): container finished" podID="836459a1-5af1-4cfc-a831-90815433a201" containerID="71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3" exitCode=0
Sep 17 17:29:52.518879 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.518868    2581 generic.go:356] "Generic (PLEG): container finished" podID="836459a1-5af1-4cfc-a831-90815433a201" containerID="4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996" exitCode=0
Sep 17 17:29:52.519118 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.518901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" event={"ID":"836459a1-5af1-4cfc-a831-90815433a201","Type":"ContainerDied","Data":"71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3"}
Sep 17 17:29:52.519118 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.518925    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"
Sep 17 17:29:52.519118 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.519008    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" event={"ID":"836459a1-5af1-4cfc-a831-90815433a201","Type":"ContainerDied","Data":"4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996"}
Sep 17 17:29:52.519118 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.519029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9" event={"ID":"836459a1-5af1-4cfc-a831-90815433a201","Type":"ContainerDied","Data":"f8f6b16a9c200ced26cb9880217da788ded564ec23166ab088648f5e8bc0819d"}
Sep 17 17:29:52.519118 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.519050    2581 scope.go:117] "RemoveContainer" containerID="71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3"
Sep 17 17:29:52.528834 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.528816    2581 scope.go:117] "RemoveContainer" containerID="957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396"
Sep 17 17:29:52.536505 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.536487    2581 scope.go:117] "RemoveContainer" containerID="4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996"
Sep 17 17:29:52.543798 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.543781    2581 scope.go:117] "RemoveContainer" containerID="71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3"
Sep 17 17:29:52.544096 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:29:52.544074    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3\": container with ID starting with 71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3 not found: ID does not exist" containerID="71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3"
Sep 17 17:29:52.544171 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.544103    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3"} err="failed to get container status \"71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3\": rpc error: code = NotFound desc = could not find container \"71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3\": container with ID starting with 71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3 not found: ID does not exist"
Sep 17 17:29:52.544171 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.544120    2581 scope.go:117] "RemoveContainer" containerID="957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396"
Sep 17 17:29:52.544387 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:29:52.544370    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396\": container with ID starting with 957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396 not found: ID does not exist" containerID="957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396"
Sep 17 17:29:52.544429 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.544394    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396"} err="failed to get container status \"957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396\": rpc error: code = NotFound desc = could not find container \"957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396\": container with ID starting with 957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396 not found: ID does not exist"
Sep 17 17:29:52.544429 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.544409    2581 scope.go:117] "RemoveContainer" containerID="4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996"
Sep 17 17:29:52.544681 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:29:52.544661    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996\": container with ID starting with 4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996 not found: ID does not exist" containerID="4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996"
Sep 17 17:29:52.544740 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.544686    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996"} err="failed to get container status \"4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996\": rpc error: code = NotFound desc = could not find container \"4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996\": container with ID starting with 4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996 not found: ID does not exist"
Sep 17 17:29:52.544740 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.544702    2581 scope.go:117] "RemoveContainer" containerID="71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3"
Sep 17 17:29:52.544938 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.544921    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3"} err="failed to get container status \"71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3\": rpc error: code = NotFound desc = could not find container \"71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3\": container with ID starting with 71d1da727afe3168ee8c22657458eaf42d7b4976244305706d1580c4f28cb1a3 not found: ID does not exist"
Sep 17 17:29:52.545008 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.544937    2581 scope.go:117] "RemoveContainer" containerID="957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396"
Sep 17 17:29:52.545211 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.545193    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396"} err="failed to get container status \"957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396\": rpc error: code = NotFound desc = could not find container \"957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396\": container with ID starting with 957fe4ceb0f61aaf7a30d2503198ce730cce24d230b998d22b907e3749961396 not found: ID does not exist"
Sep 17 17:29:52.545256 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.545211    2581 scope.go:117] "RemoveContainer" containerID="4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996"
Sep 17 17:29:52.545481 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.545460    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996"} err="failed to get container status \"4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996\": rpc error: code = NotFound desc = could not find container \"4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996\": container with ID starting with 4218ac72d3e698d08013df4392b9601d94d7b33307b7c406526e837d8bdb8996 not found: ID does not exist"
Sep 17 17:29:52.548337 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.548316    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"]
Sep 17 17:29:52.551428 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.551401    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-674fb48c94tqmg9"]
Sep 17 17:29:52.836831 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:52.836761    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="836459a1-5af1-4cfc-a831-90815433a201" path="/var/lib/kubelet/pods/836459a1-5af1-4cfc-a831-90815433a201/volumes"
Sep 17 17:29:58.773931 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:58.773880    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8" podUID="d1e9f131-eba3-4309-94d2-54cf90b0f7e4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:29:58.774497 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:29:58.774061    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"
Sep 17 17:30:08.773831 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:08.773786    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8" podUID="d1e9f131-eba3-4309-94d2-54cf90b0f7e4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:30:18.773471 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:18.773415    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8" podUID="d1e9f131-eba3-4309-94d2-54cf90b0f7e4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:30:21.390162 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.390138    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"
Sep 17 17:30:21.467943 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.467917    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1e9f131-eba3-4309-94d2-54cf90b0f7e4-tls-certs\") pod \"d1e9f131-eba3-4309-94d2-54cf90b0f7e4\" (UID: \"d1e9f131-eba3-4309-94d2-54cf90b0f7e4\") "
Sep 17 17:30:21.468080 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.467951    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n4k4\" (UniqueName: \"kubernetes.io/projected/d1e9f131-eba3-4309-94d2-54cf90b0f7e4-kube-api-access-5n4k4\") pod \"d1e9f131-eba3-4309-94d2-54cf90b0f7e4\" (UID: \"d1e9f131-eba3-4309-94d2-54cf90b0f7e4\") "
Sep 17 17:30:21.469960 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.469937    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e9f131-eba3-4309-94d2-54cf90b0f7e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1e9f131-eba3-4309-94d2-54cf90b0f7e4" (UID: "d1e9f131-eba3-4309-94d2-54cf90b0f7e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:30:21.470109 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.470089    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e9f131-eba3-4309-94d2-54cf90b0f7e4-kube-api-access-5n4k4" (OuterVolumeSpecName: "kube-api-access-5n4k4") pod "d1e9f131-eba3-4309-94d2-54cf90b0f7e4" (UID: "d1e9f131-eba3-4309-94d2-54cf90b0f7e4"). InnerVolumeSpecName "kube-api-access-5n4k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:30:21.569308 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.569284    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1e9f131-eba3-4309-94d2-54cf90b0f7e4-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:30:21.569308 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.569305    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n4k4\" (UniqueName: \"kubernetes.io/projected/d1e9f131-eba3-4309-94d2-54cf90b0f7e4-kube-api-access-5n4k4\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:30:21.659336 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.659310    2581 generic.go:356] "Generic (PLEG): container finished" podID="d1e9f131-eba3-4309-94d2-54cf90b0f7e4" containerID="4f1ca31da2f244cb94d00c9c36d1e3e798d63df6c27a3e291e1ae58e80d22b56" exitCode=0
Sep 17 17:30:21.659468 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.659370    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"
Sep 17 17:30:21.659468 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.659404    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8" event={"ID":"d1e9f131-eba3-4309-94d2-54cf90b0f7e4","Type":"ContainerDied","Data":"4f1ca31da2f244cb94d00c9c36d1e3e798d63df6c27a3e291e1ae58e80d22b56"}
Sep 17 17:30:21.659468 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.659459    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8" event={"ID":"d1e9f131-eba3-4309-94d2-54cf90b0f7e4","Type":"ContainerDied","Data":"5d273f3a1a6a4ffc454a8c9cd0ae1bfbebbc9999b6bd40b22c0d5da22f765bfd"}
Sep 17 17:30:21.659629 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.659483    2581 scope.go:117] "RemoveContainer" containerID="4f1ca31da2f244cb94d00c9c36d1e3e798d63df6c27a3e291e1ae58e80d22b56"
Sep 17 17:30:21.668840 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.668821    2581 scope.go:117] "RemoveContainer" containerID="4f1ca31da2f244cb94d00c9c36d1e3e798d63df6c27a3e291e1ae58e80d22b56"
Sep 17 17:30:21.669184 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:30:21.669162    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1ca31da2f244cb94d00c9c36d1e3e798d63df6c27a3e291e1ae58e80d22b56\": container with ID starting with 4f1ca31da2f244cb94d00c9c36d1e3e798d63df6c27a3e291e1ae58e80d22b56 not found: ID does not exist" containerID="4f1ca31da2f244cb94d00c9c36d1e3e798d63df6c27a3e291e1ae58e80d22b56"
Sep 17 17:30:21.669243 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.669194    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1ca31da2f244cb94d00c9c36d1e3e798d63df6c27a3e291e1ae58e80d22b56"} err="failed to get container status \"4f1ca31da2f244cb94d00c9c36d1e3e798d63df6c27a3e291e1ae58e80d22b56\": rpc error: code = NotFound desc = could not find container \"4f1ca31da2f244cb94d00c9c36d1e3e798d63df6c27a3e291e1ae58e80d22b56\": container with ID starting with 4f1ca31da2f244cb94d00c9c36d1e3e798d63df6c27a3e291e1ae58e80d22b56 not found: ID does not exist"
Sep 17 17:30:21.690185 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.690158    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"]
Sep 17 17:30:21.692776 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:21.692752    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched95c8"]
Sep 17 17:30:22.837104 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:22.837064    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1e9f131-eba3-4309-94d2-54cf90b0f7e4" path="/var/lib/kubelet/pods/d1e9f131-eba3-4309-94d2-54cf90b0f7e4/volumes"
Sep 17 17:30:36.942195 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.942149    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"]
Sep 17 17:30:36.942701 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.942640    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e9f131-eba3-4309-94d2-54cf90b0f7e4" containerName="main"
Sep 17 17:30:36.942701 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.942661    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e9f131-eba3-4309-94d2-54cf90b0f7e4" containerName="main"
Sep 17 17:30:36.942701 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.942681    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="storage-initializer"
Sep 17 17:30:36.942701 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.942690    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="storage-initializer"
Sep 17 17:30:36.942946 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.942720    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main"
Sep 17 17:30:36.942946 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.942729    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main"
Sep 17 17:30:36.942946 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.942739    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="llm-d-routing-sidecar"
Sep 17 17:30:36.942946 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.942747    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="llm-d-routing-sidecar"
Sep 17 17:30:36.942946 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.942857    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="main"
Sep 17 17:30:36.942946 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.942873    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="836459a1-5af1-4cfc-a831-90815433a201" containerName="llm-d-routing-sidecar"
Sep 17 17:30:36.942946 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.942883    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e9f131-eba3-4309-94d2-54cf90b0f7e4" containerName="main"
Sep 17 17:30:36.946286 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.946260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:36.951188 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.951163    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 17:30:36.956148 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.956123    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 17:30:36.956265 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.956154    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-gmzjw\""
Sep 17 17:30:36.956265 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.956172    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 17:30:36.956265 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.956174    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:30:36.956458 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.956444    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 17:30:36.958062 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.958044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-rz8kz\""
Sep 17 17:30:36.999622 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:36.999596    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"]
Sep 17 17:30:37.082717 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.082693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-model-cache\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.082828 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.082734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.082828 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.082757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.082828 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.082810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-dshm\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.083052 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.082836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.083052 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.082854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.083052 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.082900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-home\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.083052 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.082991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kpv4\" (UniqueName: \"kubernetes.io/projected/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-kube-api-access-7kpv4\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.184024 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.183996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kpv4\" (UniqueName: \"kubernetes.io/projected/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-kube-api-access-7kpv4\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.184129 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.184037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-model-cache\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.184129 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.184090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.184129 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.184126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.184281 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.184151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-dshm\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.184281 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.184169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.184281 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.184186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.184281 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.184213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-home\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.184575 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.184543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.184647 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.184597    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.184694 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.184673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-model-cache\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.184792 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.184758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-home\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.185035 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.185013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.186733 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.186714    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-dshm\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.187042 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.187020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.213155 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.213097    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kpv4\" (UniqueName: \"kubernetes.io/projected/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-kube-api-access-7kpv4\") pod \"custom-route-timeout-pd-test-kserve-68475bb755-psxxj\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.255909 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.255886    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.337561 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.337529    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"]
Sep 17 17:30:37.341611 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.341591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"
Sep 17 17:30:37.345089 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.345069    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-ps66p\""
Sep 17 17:30:37.357587 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.357565    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"]
Sep 17 17:30:37.486317 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.486254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/796651f3-8e63-4e54-8c88-4dceaab4aadd-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj\" (UID: \"796651f3-8e63-4e54-8c88-4dceaab4aadd\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"
Sep 17 17:30:37.486317 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.486297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjfjx\" (UniqueName: \"kubernetes.io/projected/796651f3-8e63-4e54-8c88-4dceaab4aadd-kube-api-access-sjfjx\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj\" (UID: \"796651f3-8e63-4e54-8c88-4dceaab4aadd\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"
Sep 17 17:30:37.549325 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.549303    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"]
Sep 17 17:30:37.551415 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:30:37.551392    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9f4cc7_0da5_4736_95ff_03b8bb01b8de.slice/crio-31c5a9d9a8ae394da66450c519af46f0050e2f71f9fbe81c5de9fedfbed39ca3 WatchSource:0}: Error finding container 31c5a9d9a8ae394da66450c519af46f0050e2f71f9fbe81c5de9fedfbed39ca3: Status 404 returned error can't find the container with id 31c5a9d9a8ae394da66450c519af46f0050e2f71f9fbe81c5de9fedfbed39ca3
Sep 17 17:30:37.586695 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.586673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjfjx\" (UniqueName: \"kubernetes.io/projected/796651f3-8e63-4e54-8c88-4dceaab4aadd-kube-api-access-sjfjx\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj\" (UID: \"796651f3-8e63-4e54-8c88-4dceaab4aadd\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"
Sep 17 17:30:37.586800 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.586736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/796651f3-8e63-4e54-8c88-4dceaab4aadd-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj\" (UID: \"796651f3-8e63-4e54-8c88-4dceaab4aadd\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"
Sep 17 17:30:37.589033 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.589012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/796651f3-8e63-4e54-8c88-4dceaab4aadd-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj\" (UID: \"796651f3-8e63-4e54-8c88-4dceaab4aadd\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"
Sep 17 17:30:37.596383 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.596299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjfjx\" (UniqueName: \"kubernetes.io/projected/796651f3-8e63-4e54-8c88-4dceaab4aadd-kube-api-access-sjfjx\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj\" (UID: \"796651f3-8e63-4e54-8c88-4dceaab4aadd\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"
Sep 17 17:30:37.652808 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.652787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"
Sep 17 17:30:37.738019 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.737940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" event={"ID":"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de","Type":"ContainerStarted","Data":"f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723"}
Sep 17 17:30:37.738019 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.738011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" event={"ID":"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de","Type":"ContainerStarted","Data":"31c5a9d9a8ae394da66450c519af46f0050e2f71f9fbe81c5de9fedfbed39ca3"}
Sep 17 17:30:37.738194 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.738089    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:37.929354 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:37.929327    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"]
Sep 17 17:30:37.930382 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:30:37.930356    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796651f3_8e63_4e54_8c88_4dceaab4aadd.slice/crio-3ed0659ad36b9bf7ddd3adae871b975b19d031794e8cf2b22b6c94f7f0a8e1e2 WatchSource:0}: Error finding container 3ed0659ad36b9bf7ddd3adae871b975b19d031794e8cf2b22b6c94f7f0a8e1e2: Status 404 returned error can't find the container with id 3ed0659ad36b9bf7ddd3adae871b975b19d031794e8cf2b22b6c94f7f0a8e1e2
Sep 17 17:30:38.743767 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:38.743735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj" event={"ID":"796651f3-8e63-4e54-8c88-4dceaab4aadd","Type":"ContainerStarted","Data":"1e9aa43da187d0c9f3a28b2e09b5f969f62abb0d95678a51744e6916d15533b2"}
Sep 17 17:30:38.744266 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:38.743784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj" event={"ID":"796651f3-8e63-4e54-8c88-4dceaab4aadd","Type":"ContainerStarted","Data":"3ed0659ad36b9bf7ddd3adae871b975b19d031794e8cf2b22b6c94f7f0a8e1e2"}
Sep 17 17:30:38.744266 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:38.743917    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"
Sep 17 17:30:38.745830 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:38.745801    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" event={"ID":"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de","Type":"ContainerStarted","Data":"d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e"}
Sep 17 17:30:38.807849 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:38.807800    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj" podStartSLOduration=1.807788854 podStartE2EDuration="1.807788854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:30:38.790338716 +0000 UTC m=+2918.213276578" watchObservedRunningTime="2026-09-17 17:30:38.807788854 +0000 UTC m=+2918.230726716"
Sep 17 17:30:40.756911 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:40.756872    2581 generic.go:356] "Generic (PLEG): container finished" podID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerID="d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e" exitCode=0
Sep 17 17:30:40.757330 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:40.756947    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" event={"ID":"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de","Type":"ContainerDied","Data":"d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e"}
Sep 17 17:30:41.763388 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:41.763353    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" event={"ID":"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de","Type":"ContainerStarted","Data":"cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e"}
Sep 17 17:30:41.794725 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:41.794666    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" podStartSLOduration=3.62971309 podStartE2EDuration="5.794644262s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.164931172s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:30:41.79139905 +0000 UTC m=+2921.214336914" watchObservedRunningTime="2026-09-17 17:30:41.794644262 +0000 UTC m=+2921.217582126"
Sep 17 17:30:47.256498 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:47.256455    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:47.257027 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:47.256514    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:47.257778 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:47.257747    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Sep 17 17:30:47.276858 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:47.276835    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:30:57.256983 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:30:57.256918    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Sep 17 17:31:07.256520 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:31:07.256472    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Sep 17 17:31:09.752707 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:31:09.752675    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"
Sep 17 17:31:17.256737 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:31:17.256643    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Sep 17 17:31:27.256499 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:31:27.256445    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Sep 17 17:31:37.256566 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:31:37.256513    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Sep 17 17:31:47.256372 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:31:47.256327    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Sep 17 17:31:57.257235 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:31:57.257186    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Sep 17 17:32:02.971674 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:32:02.971645    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:32:02.980981 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:32:02.980946    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:32:07.256819 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:32:07.256772    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Sep 17 17:32:17.257335 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:32:17.257223    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Sep 17 17:32:27.266724 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:32:27.266683    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:32:27.283126 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:32:27.283096    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:32:57.187847 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:32:57.187764    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"]
Sep 17 17:32:57.192454 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:32:57.192426    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"]
Sep 17 17:32:59.752324 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:32:59.752283    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj" podUID="796651f3-8e63-4e54-8c88-4dceaab4aadd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:33:09.753031 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:09.752987    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj" podUID="796651f3-8e63-4e54-8c88-4dceaab4aadd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:33:12.208248 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:12.208190    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj" podUID="796651f3-8e63-4e54-8c88-4dceaab4aadd" containerName="main" containerID="cri-o://1e9aa43da187d0c9f3a28b2e09b5f969f62abb0d95678a51744e6916d15533b2" gracePeriod=45
Sep 17 17:33:12.214870 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:12.214839    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="main" containerID="cri-o://cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e" gracePeriod=45
Sep 17 17:33:12.831895 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:12.831840    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="llm-d-routing-sidecar" containerID="cri-o://f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723" gracePeriod=45
Sep 17 17:33:13.068592 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.068568    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:33:13.138342 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.138316    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-kserve-provision-location\") pod \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") "
Sep 17 17:33:13.138460 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.138352    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-tls-certs\") pod \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") "
Sep 17 17:33:13.138460 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.138374    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-model-cache\") pod \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") "
Sep 17 17:33:13.138537 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.138495    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kpv4\" (UniqueName: \"kubernetes.io/projected/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-kube-api-access-7kpv4\") pod \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") "
Sep 17 17:33:13.138570 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.138542    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-home\") pod \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") "
Sep 17 17:33:13.138612 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.138584    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-cabundle-cert\") pod \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") "
Sep 17 17:33:13.138658 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.138623    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" (UID: "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:33:13.138658 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.138640    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-tmp-dir\") pod \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") "
Sep 17 17:33:13.138773 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.138664    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-dshm\") pod \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\" (UID: \"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de\") "
Sep 17 17:33:13.139007 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.138940    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" (UID: "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:33:13.139007 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.138953    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.139401 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.139368    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-home" (OuterVolumeSpecName: "home") pod "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" (UID: "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:33:13.140853 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.140818    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-kube-api-access-7kpv4" (OuterVolumeSpecName: "kube-api-access-7kpv4") pod "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" (UID: "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de"). InnerVolumeSpecName "kube-api-access-7kpv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:33:13.140853 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.140829    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-dshm" (OuterVolumeSpecName: "dshm") pod "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" (UID: "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:33:13.141100 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.141081    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" (UID: "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:33:13.150693 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.150669    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" (UID: "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:33:13.162246 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.162221    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" (UID: "8d9f4cc7-0da5-4736-95ff-03b8bb01b8de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:33:13.239576 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.239553    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kpv4\" (UniqueName: \"kubernetes.io/projected/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-kube-api-access-7kpv4\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.239576 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.239574    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.239908 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.239583    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.239908 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.239610    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.239908 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.239618    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.239908 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.239626    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.239908 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.239634    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:33:13.509239 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.509203    2581 generic.go:356] "Generic (PLEG): container finished" podID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerID="cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e" exitCode=0
Sep 17 17:33:13.509239 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.509235    2581 generic.go:356] "Generic (PLEG): container finished" podID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerID="f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723" exitCode=0
Sep 17 17:33:13.509412 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.509277    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"
Sep 17 17:33:13.509412 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.509294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" event={"ID":"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de","Type":"ContainerDied","Data":"cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e"}
Sep 17 17:33:13.509412 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.509360    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" event={"ID":"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de","Type":"ContainerDied","Data":"f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723"}
Sep 17 17:33:13.509412 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.509381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj" event={"ID":"8d9f4cc7-0da5-4736-95ff-03b8bb01b8de","Type":"ContainerDied","Data":"31c5a9d9a8ae394da66450c519af46f0050e2f71f9fbe81c5de9fedfbed39ca3"}
Sep 17 17:33:13.509412 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.509404    2581 scope.go:117] "RemoveContainer" containerID="cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e"
Sep 17 17:33:13.519866 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.519845    2581 scope.go:117] "RemoveContainer" containerID="d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e"
Sep 17 17:33:13.528384 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.528365    2581 scope.go:117] "RemoveContainer" containerID="f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723"
Sep 17 17:33:13.536140 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.536124    2581 scope.go:117] "RemoveContainer" containerID="cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e"
Sep 17 17:33:13.536415 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:33:13.536393    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e\": container with ID starting with cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e not found: ID does not exist" containerID="cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e"
Sep 17 17:33:13.536515 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.536423    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e"} err="failed to get container status \"cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e\": rpc error: code = NotFound desc = could not find container \"cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e\": container with ID starting with cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e not found: ID does not exist"
Sep 17 17:33:13.536515 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.536441    2581 scope.go:117] "RemoveContainer" containerID="d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e"
Sep 17 17:33:13.536828 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:33:13.536794    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e\": container with ID starting with d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e not found: ID does not exist" containerID="d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e"
Sep 17 17:33:13.537016 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.536834    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e"} err="failed to get container status \"d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e\": rpc error: code = NotFound desc = could not find container \"d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e\": container with ID starting with d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e not found: ID does not exist"
Sep 17 17:33:13.537016 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.536858    2581 scope.go:117] "RemoveContainer" containerID="f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723"
Sep 17 17:33:13.537475 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:33:13.537446    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723\": container with ID starting with f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723 not found: ID does not exist" containerID="f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723"
Sep 17 17:33:13.537641 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.537482    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723"} err="failed to get container status \"f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723\": rpc error: code = NotFound desc = could not find container \"f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723\": container with ID starting with f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723 not found: ID does not exist"
Sep 17 17:33:13.537641 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.537500    2581 scope.go:117] "RemoveContainer" containerID="cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e"
Sep 17 17:33:13.537863 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.537842    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e"} err="failed to get container status \"cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e\": rpc error: code = NotFound desc = could not find container \"cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e\": container with ID starting with cbf15092cdfdaa03bbc507c79368f91cbf46b612c7f23f0892c3827dec9d619e not found: ID does not exist"
Sep 17 17:33:13.537924 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.537865    2581 scope.go:117] "RemoveContainer" containerID="d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e"
Sep 17 17:33:13.538292 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.538259    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e"} err="failed to get container status \"d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e\": rpc error: code = NotFound desc = could not find container \"d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e\": container with ID starting with d6847d4a8add7e4213d5ad24c3d80f4ecc9702819afeee87a93808f161411a4e not found: ID does not exist"
Sep 17 17:33:13.538292 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.538289    2581 scope.go:117] "RemoveContainer" containerID="f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723"
Sep 17 17:33:13.538713 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.538685    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723"} err="failed to get container status \"f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723\": rpc error: code = NotFound desc = could not find container \"f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723\": container with ID starting with f3618513ec0180d20e1bb1a9d700fb07559bce7a0a1a9c4d961605792064d723 not found: ID does not exist"
Sep 17 17:33:13.538830 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.538775    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"]
Sep 17 17:33:13.541365 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:13.541343    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-68475bb755-psxxj"]
Sep 17 17:33:14.836200 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:14.836149    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" path="/var/lib/kubelet/pods/8d9f4cc7-0da5-4736-95ff-03b8bb01b8de/volumes"
Sep 17 17:33:19.752474 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:19.752431    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj" podUID="796651f3-8e63-4e54-8c88-4dceaab4aadd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:33:19.752923 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:19.752531    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"
Sep 17 17:33:29.752677 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:29.752634    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj" podUID="796651f3-8e63-4e54-8c88-4dceaab4aadd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:33:39.752856 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:39.752806    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj" podUID="796651f3-8e63-4e54-8c88-4dceaab4aadd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:33:42.449641 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.449617    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"
Sep 17 17:33:42.559380 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.559319    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/796651f3-8e63-4e54-8c88-4dceaab4aadd-tls-certs\") pod \"796651f3-8e63-4e54-8c88-4dceaab4aadd\" (UID: \"796651f3-8e63-4e54-8c88-4dceaab4aadd\") "
Sep 17 17:33:42.559380 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.559377    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjfjx\" (UniqueName: \"kubernetes.io/projected/796651f3-8e63-4e54-8c88-4dceaab4aadd-kube-api-access-sjfjx\") pod \"796651f3-8e63-4e54-8c88-4dceaab4aadd\" (UID: \"796651f3-8e63-4e54-8c88-4dceaab4aadd\") "
Sep 17 17:33:42.561406 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.561377    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796651f3-8e63-4e54-8c88-4dceaab4aadd-kube-api-access-sjfjx" (OuterVolumeSpecName: "kube-api-access-sjfjx") pod "796651f3-8e63-4e54-8c88-4dceaab4aadd" (UID: "796651f3-8e63-4e54-8c88-4dceaab4aadd"). InnerVolumeSpecName "kube-api-access-sjfjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:33:42.561503 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.561482    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/796651f3-8e63-4e54-8c88-4dceaab4aadd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "796651f3-8e63-4e54-8c88-4dceaab4aadd" (UID: "796651f3-8e63-4e54-8c88-4dceaab4aadd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:33:42.645486 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.645460    2581 generic.go:356] "Generic (PLEG): container finished" podID="796651f3-8e63-4e54-8c88-4dceaab4aadd" containerID="1e9aa43da187d0c9f3a28b2e09b5f969f62abb0d95678a51744e6916d15533b2" exitCode=0
Sep 17 17:33:42.645591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.645501    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"
Sep 17 17:33:42.645591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.645511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj" event={"ID":"796651f3-8e63-4e54-8c88-4dceaab4aadd","Type":"ContainerDied","Data":"1e9aa43da187d0c9f3a28b2e09b5f969f62abb0d95678a51744e6916d15533b2"}
Sep 17 17:33:42.645591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.645553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj" event={"ID":"796651f3-8e63-4e54-8c88-4dceaab4aadd","Type":"ContainerDied","Data":"3ed0659ad36b9bf7ddd3adae871b975b19d031794e8cf2b22b6c94f7f0a8e1e2"}
Sep 17 17:33:42.645591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.645573    2581 scope.go:117] "RemoveContainer" containerID="1e9aa43da187d0c9f3a28b2e09b5f969f62abb0d95678a51744e6916d15533b2"
Sep 17 17:33:42.653930 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.653914    2581 scope.go:117] "RemoveContainer" containerID="1e9aa43da187d0c9f3a28b2e09b5f969f62abb0d95678a51744e6916d15533b2"
Sep 17 17:33:42.654210 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:33:42.654189    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e9aa43da187d0c9f3a28b2e09b5f969f62abb0d95678a51744e6916d15533b2\": container with ID starting with 1e9aa43da187d0c9f3a28b2e09b5f969f62abb0d95678a51744e6916d15533b2 not found: ID does not exist" containerID="1e9aa43da187d0c9f3a28b2e09b5f969f62abb0d95678a51744e6916d15533b2"
Sep 17 17:33:42.654270 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.654217    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e9aa43da187d0c9f3a28b2e09b5f969f62abb0d95678a51744e6916d15533b2"} err="failed to get container status \"1e9aa43da187d0c9f3a28b2e09b5f969f62abb0d95678a51744e6916d15533b2\": rpc error: code = NotFound desc = could not find container \"1e9aa43da187d0c9f3a28b2e09b5f969f62abb0d95678a51744e6916d15533b2\": container with ID starting with 1e9aa43da187d0c9f3a28b2e09b5f969f62abb0d95678a51744e6916d15533b2 not found: ID does not exist"
Sep 17 17:33:42.660619 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.660598    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/796651f3-8e63-4e54-8c88-4dceaab4aadd-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:33:42.660697 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.660622    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjfjx\" (UniqueName: \"kubernetes.io/projected/796651f3-8e63-4e54-8c88-4dceaab4aadd-kube-api-access-sjfjx\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:33:42.668430 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.668411    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"]
Sep 17 17:33:42.674816 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.674798    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c486zxrj"]
Sep 17 17:33:42.835652 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:33:42.835606    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="796651f3-8e63-4e54-8c88-4dceaab4aadd" path="/var/lib/kubelet/pods/796651f3-8e63-4e54-8c88-4dceaab4aadd/volumes"
Sep 17 17:34:02.308842 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.308805    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"]
Sep 17 17:34:02.309311 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.309127    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="llm-d-routing-sidecar"
Sep 17 17:34:02.309311 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.309139    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="llm-d-routing-sidecar"
Sep 17 17:34:02.309311 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.309148    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="796651f3-8e63-4e54-8c88-4dceaab4aadd" containerName="main"
Sep 17 17:34:02.309311 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.309153    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="796651f3-8e63-4e54-8c88-4dceaab4aadd" containerName="main"
Sep 17 17:34:02.309311 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.309162    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="storage-initializer"
Sep 17 17:34:02.309311 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.309167    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="storage-initializer"
Sep 17 17:34:02.309311 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.309178    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="main"
Sep 17 17:34:02.309311 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.309183    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="main"
Sep 17 17:34:02.309311 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.309253    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="llm-d-routing-sidecar"
Sep 17 17:34:02.309311 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.309263    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9f4cc7-0da5-4736-95ff-03b8bb01b8de" containerName="main"
Sep 17 17:34:02.309311 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.309271    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="796651f3-8e63-4e54-8c88-4dceaab4aadd" containerName="main"
Sep 17 17:34:02.313811 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.313789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.319630 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.319608    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 17:34:02.319756 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.319705    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-svcd9\""
Sep 17 17:34:02.319860 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.319843    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 17:34:02.321363 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.321334    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:34:02.321495 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.321413    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 17:34:02.323112 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.323088    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 17:34:02.328407 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.328393    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-ggld5\""
Sep 17 17:34:02.347380 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.347344    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"]
Sep 17 17:34:02.409462 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.409424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkjx8\" (UniqueName: \"kubernetes.io/projected/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-kube-api-access-kkjx8\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.409462 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.409460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.409675 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.409480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-model-cache\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.409675 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.409504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.409675 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.409569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-dshm\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.409675 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.409598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-tls-certs\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.409675 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.409638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-home\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.409839 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.409691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.510214 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.510175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-home\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.510400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.510222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.510400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.510268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjx8\" (UniqueName: \"kubernetes.io/projected/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-kube-api-access-kkjx8\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.510400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.510291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.510400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.510307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-model-cache\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.510400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.510323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.510400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.510366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-dshm\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.510400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.510391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-tls-certs\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.510711 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.510668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-home\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.510770 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.510713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-model-cache\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.510770 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.510738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.510878 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.510800    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.511093 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.511071    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.512926 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.512906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-dshm\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.513087 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.513071    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-tls-certs\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.539935 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.539899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkjx8\" (UniqueName: \"kubernetes.io/projected/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-kube-api-access-kkjx8\") pod \"router-with-refs-pd-test-kserve-6b557568fc-7bd2z\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.623775 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.623681    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:02.921536 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:02.921505    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"]
Sep 17 17:34:02.924119 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:34:02.924090    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ddc72bf_f59e_47b9_ab06_56b3370a3f66.slice/crio-948f11c788a0a054789b69a2b5c95e74084c112926d4c7856cb1aa2a8bc6606a WatchSource:0}: Error finding container 948f11c788a0a054789b69a2b5c95e74084c112926d4c7856cb1aa2a8bc6606a: Status 404 returned error can't find the container with id 948f11c788a0a054789b69a2b5c95e74084c112926d4c7856cb1aa2a8bc6606a
Sep 17 17:34:03.001429 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.001399    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"]
Sep 17 17:34:03.004953 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.004921    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"
Sep 17 17:34:03.008003 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.007984    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-494t2\""
Sep 17 17:34:03.014448 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.014417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjftm\" (UniqueName: \"kubernetes.io/projected/58b0fb12-23e0-4de8-b639-46d4045ba4c4-kube-api-access-tjftm\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9\" (UID: \"58b0fb12-23e0-4de8-b639-46d4045ba4c4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"
Sep 17 17:34:03.014559 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.014468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b0fb12-23e0-4de8-b639-46d4045ba4c4-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9\" (UID: \"58b0fb12-23e0-4de8-b639-46d4045ba4c4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"
Sep 17 17:34:03.024249 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.024228    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"]
Sep 17 17:34:03.115302 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.115271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjftm\" (UniqueName: \"kubernetes.io/projected/58b0fb12-23e0-4de8-b639-46d4045ba4c4-kube-api-access-tjftm\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9\" (UID: \"58b0fb12-23e0-4de8-b639-46d4045ba4c4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"
Sep 17 17:34:03.115472 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.115308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b0fb12-23e0-4de8-b639-46d4045ba4c4-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9\" (UID: \"58b0fb12-23e0-4de8-b639-46d4045ba4c4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"
Sep 17 17:34:03.117858 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.117826    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b0fb12-23e0-4de8-b639-46d4045ba4c4-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9\" (UID: \"58b0fb12-23e0-4de8-b639-46d4045ba4c4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"
Sep 17 17:34:03.130285 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.130255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjftm\" (UniqueName: \"kubernetes.io/projected/58b0fb12-23e0-4de8-b639-46d4045ba4c4-kube-api-access-tjftm\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9\" (UID: \"58b0fb12-23e0-4de8-b639-46d4045ba4c4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"
Sep 17 17:34:03.315309 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.315274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"
Sep 17 17:34:03.597409 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.597330    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"]
Sep 17 17:34:03.600763 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:34:03.600732    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b0fb12_23e0_4de8_b639_46d4045ba4c4.slice/crio-5218f7cd36735151e0878f69ab63caa0a2f6ff8f746fdb03cc21d370ebeff589 WatchSource:0}: Error finding container 5218f7cd36735151e0878f69ab63caa0a2f6ff8f746fdb03cc21d370ebeff589: Status 404 returned error can't find the container with id 5218f7cd36735151e0878f69ab63caa0a2f6ff8f746fdb03cc21d370ebeff589
Sep 17 17:34:03.749588 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.749543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" event={"ID":"4ddc72bf-f59e-47b9-ab06-56b3370a3f66","Type":"ContainerStarted","Data":"4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0"}
Sep 17 17:34:03.749783 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.749665    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:03.749783 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.749699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" event={"ID":"4ddc72bf-f59e-47b9-ab06-56b3370a3f66","Type":"ContainerStarted","Data":"948f11c788a0a054789b69a2b5c95e74084c112926d4c7856cb1aa2a8bc6606a"}
Sep 17 17:34:03.751321 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.751292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9" event={"ID":"58b0fb12-23e0-4de8-b639-46d4045ba4c4","Type":"ContainerStarted","Data":"9f307c6276753b376adc542d3711ce04ef3eab7895a1aeaa4e106f773edbc57b"}
Sep 17 17:34:03.751440 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.751347    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"
Sep 17 17:34:03.751440 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.751365    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9" event={"ID":"58b0fb12-23e0-4de8-b639-46d4045ba4c4","Type":"ContainerStarted","Data":"5218f7cd36735151e0878f69ab63caa0a2f6ff8f746fdb03cc21d370ebeff589"}
Sep 17 17:34:03.753824 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.753802    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 17:34:03.816639 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:03.816453    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9" podStartSLOduration=1.816437304 podStartE2EDuration="1.816437304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:03.815089112 +0000 UTC m=+3123.238026979" watchObservedRunningTime="2026-09-17 17:34:03.816437304 +0000 UTC m=+3123.239375167"
Sep 17 17:34:04.757337 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:04.757296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" event={"ID":"4ddc72bf-f59e-47b9-ab06-56b3370a3f66","Type":"ContainerStarted","Data":"b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d"}
Sep 17 17:34:06.769776 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:06.769742    2581 generic.go:356] "Generic (PLEG): container finished" podID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerID="b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d" exitCode=0
Sep 17 17:34:06.770146 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:06.769816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" event={"ID":"4ddc72bf-f59e-47b9-ab06-56b3370a3f66","Type":"ContainerDied","Data":"b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d"}
Sep 17 17:34:07.777755 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:07.777718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" event={"ID":"4ddc72bf-f59e-47b9-ab06-56b3370a3f66","Type":"ContainerStarted","Data":"14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1"}
Sep 17 17:34:07.807556 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:07.807495    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" podStartSLOduration=3.609836874 podStartE2EDuration="5.807472142s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.197635268s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:07.805872134 +0000 UTC m=+3127.228809997" watchObservedRunningTime="2026-09-17 17:34:07.807472142 +0000 UTC m=+3127.230410006"
Sep 17 17:34:12.624255 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:12.624212    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:12.624699 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:12.624292    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:12.625698 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:12.625655    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Sep 17 17:34:12.637570 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:12.637547    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:34:22.624532 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:22.624433    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Sep 17 17:34:32.624751 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:32.624695    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Sep 17 17:34:34.759598 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:34.759567    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"
Sep 17 17:34:42.624892 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:42.624841    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Sep 17 17:34:52.624581 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:34:52.624533    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Sep 17 17:35:02.624330 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:35:02.624279    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Sep 17 17:35:12.624187 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:35:12.624140    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Sep 17 17:35:22.624721 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:35:22.624668    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Sep 17 17:35:32.624582 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:35:32.624524    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Sep 17 17:35:42.624284 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:35:42.624222    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Sep 17 17:35:52.624844 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:35:52.624724    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Sep 17 17:36:02.634400 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:02.634366    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:36:02.650558 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:02.650527    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:36:22.453776 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:22.453741    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"]
Sep 17 17:36:22.462108 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:22.462079    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"]
Sep 17 17:36:24.759431 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:24.759394    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9" podUID="58b0fb12-23e0-4de8-b639-46d4045ba4c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:36:34.759033 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:34.758989    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9" podUID="58b0fb12-23e0-4de8-b639-46d4045ba4c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:36:37.476811 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:37.476765    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main" containerID="cri-o://14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1" gracePeriod=45
Sep 17 17:36:37.480167 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:37.480136    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9" podUID="58b0fb12-23e0-4de8-b639-46d4045ba4c4" containerName="main" containerID="cri-o://9f307c6276753b376adc542d3711ce04ef3eab7895a1aeaa4e106f773edbc57b" gracePeriod=45
Sep 17 17:36:38.088622 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.088567    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="llm-d-routing-sidecar" containerID="cri-o://4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0" gracePeriod=45
Sep 17 17:36:38.328802 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.328776    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:36:38.368315 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.368288    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-dshm\") pod \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") "
Sep 17 17:36:38.368504 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.368322    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-tls-certs\") pod \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") "
Sep 17 17:36:38.368504 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.368354    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-home\") pod \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") "
Sep 17 17:36:38.368504 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.368373    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-model-cache\") pod \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") "
Sep 17 17:36:38.368504 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.368399    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-kserve-provision-location\") pod \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") "
Sep 17 17:36:38.368504 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.368433    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-tmp-dir\") pod \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") "
Sep 17 17:36:38.368504 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.368484    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-cabundle-cert\") pod \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") "
Sep 17 17:36:38.368863 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.368533    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkjx8\" (UniqueName: \"kubernetes.io/projected/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-kube-api-access-kkjx8\") pod \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\" (UID: \"4ddc72bf-f59e-47b9-ab06-56b3370a3f66\") "
Sep 17 17:36:38.368863 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.368646    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ddc72bf-f59e-47b9-ab06-56b3370a3f66" (UID: "4ddc72bf-f59e-47b9-ab06-56b3370a3f66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:38.368863 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.368769    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.369064 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.368982    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ddc72bf-f59e-47b9-ab06-56b3370a3f66" (UID: "4ddc72bf-f59e-47b9-ab06-56b3370a3f66"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:38.369177 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.369148    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-home" (OuterVolumeSpecName: "home") pod "4ddc72bf-f59e-47b9-ab06-56b3370a3f66" (UID: "4ddc72bf-f59e-47b9-ab06-56b3370a3f66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:38.371122 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.371094    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-dshm" (OuterVolumeSpecName: "dshm") pod "4ddc72bf-f59e-47b9-ab06-56b3370a3f66" (UID: "4ddc72bf-f59e-47b9-ab06-56b3370a3f66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:38.371216 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.371136    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ddc72bf-f59e-47b9-ab06-56b3370a3f66" (UID: "4ddc72bf-f59e-47b9-ab06-56b3370a3f66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:38.371263 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.371218    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-kube-api-access-kkjx8" (OuterVolumeSpecName: "kube-api-access-kkjx8") pod "4ddc72bf-f59e-47b9-ab06-56b3370a3f66" (UID: "4ddc72bf-f59e-47b9-ab06-56b3370a3f66"). InnerVolumeSpecName "kube-api-access-kkjx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:38.385340 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.385289    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ddc72bf-f59e-47b9-ab06-56b3370a3f66" (UID: "4ddc72bf-f59e-47b9-ab06-56b3370a3f66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:38.397595 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.397564    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ddc72bf-f59e-47b9-ab06-56b3370a3f66" (UID: "4ddc72bf-f59e-47b9-ab06-56b3370a3f66"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:38.469351 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.469319    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.469351 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.469349    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.469351 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.469359    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.469577 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.469369    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.469577 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.469378    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.469577 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.469386    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.469577 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.469395    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkjx8\" (UniqueName: \"kubernetes.io/projected/4ddc72bf-f59e-47b9-ab06-56b3370a3f66-kube-api-access-kkjx8\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:36:38.517606 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.517523    2581 generic.go:356] "Generic (PLEG): container finished" podID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerID="14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1" exitCode=0
Sep 17 17:36:38.517606 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.517548    2581 generic.go:356] "Generic (PLEG): container finished" podID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerID="4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0" exitCode=0
Sep 17 17:36:38.517606 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.517577    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"
Sep 17 17:36:38.517606 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.517592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" event={"ID":"4ddc72bf-f59e-47b9-ab06-56b3370a3f66","Type":"ContainerDied","Data":"14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1"}
Sep 17 17:36:38.518169 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.517634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" event={"ID":"4ddc72bf-f59e-47b9-ab06-56b3370a3f66","Type":"ContainerDied","Data":"4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0"}
Sep 17 17:36:38.518169 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.517650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z" event={"ID":"4ddc72bf-f59e-47b9-ab06-56b3370a3f66","Type":"ContainerDied","Data":"948f11c788a0a054789b69a2b5c95e74084c112926d4c7856cb1aa2a8bc6606a"}
Sep 17 17:36:38.518169 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.517672    2581 scope.go:117] "RemoveContainer" containerID="14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1"
Sep 17 17:36:38.526250 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.526232    2581 scope.go:117] "RemoveContainer" containerID="b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d"
Sep 17 17:36:38.533728 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.533712    2581 scope.go:117] "RemoveContainer" containerID="4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0"
Sep 17 17:36:38.540712 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.540694    2581 scope.go:117] "RemoveContainer" containerID="14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1"
Sep 17 17:36:38.540955 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:36:38.540927    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1\": container with ID starting with 14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1 not found: ID does not exist" containerID="14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1"
Sep 17 17:36:38.541067 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.540958    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1"} err="failed to get container status \"14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1\": rpc error: code = NotFound desc = could not find container \"14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1\": container with ID starting with 14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1 not found: ID does not exist"
Sep 17 17:36:38.541067 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.541057    2581 scope.go:117] "RemoveContainer" containerID="b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d"
Sep 17 17:36:38.541325 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:36:38.541307    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d\": container with ID starting with b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d not found: ID does not exist" containerID="b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d"
Sep 17 17:36:38.541392 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.541346    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d"} err="failed to get container status \"b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d\": rpc error: code = NotFound desc = could not find container \"b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d\": container with ID starting with b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d not found: ID does not exist"
Sep 17 17:36:38.541392 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.541361    2581 scope.go:117] "RemoveContainer" containerID="4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0"
Sep 17 17:36:38.541589 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:36:38.541569    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0\": container with ID starting with 4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0 not found: ID does not exist" containerID="4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0"
Sep 17 17:36:38.541653 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.541596    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0"} err="failed to get container status \"4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0\": rpc error: code = NotFound desc = could not find container \"4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0\": container with ID starting with 4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0 not found: ID does not exist"
Sep 17 17:36:38.541653 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.541620    2581 scope.go:117] "RemoveContainer" containerID="14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1"
Sep 17 17:36:38.541887 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.541870    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1"} err="failed to get container status \"14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1\": rpc error: code = NotFound desc = could not find container \"14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1\": container with ID starting with 14e32c316169f7556239a79a3cb9c0bcc67ffde1d8c29940d5e0603fa627e7a1 not found: ID does not exist"
Sep 17 17:36:38.541887 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.541886    2581 scope.go:117] "RemoveContainer" containerID="b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d"
Sep 17 17:36:38.542103 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.542083    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d"} err="failed to get container status \"b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d\": rpc error: code = NotFound desc = could not find container \"b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d\": container with ID starting with b37a75fe900bdb6a34071dd58e6d3e7a170243783d4d7ca98621caf56c72182d not found: ID does not exist"
Sep 17 17:36:38.542155 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.542104    2581 scope.go:117] "RemoveContainer" containerID="4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0"
Sep 17 17:36:38.542337 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.542318    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0"} err="failed to get container status \"4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0\": rpc error: code = NotFound desc = could not find container \"4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0\": container with ID starting with 4b65fc0b20a096f63361dd4cf97abc6e8620fb998ebd848de453557e8f5263c0 not found: ID does not exist"
Sep 17 17:36:38.546082 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.546061    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"]
Sep 17 17:36:38.551915 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.551894    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6b557568fc-7bd2z"]
Sep 17 17:36:38.835978 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:38.835887    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" path="/var/lib/kubelet/pods/4ddc72bf-f59e-47b9-ab06-56b3370a3f66/volumes"
Sep 17 17:36:44.759406 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:44.759363    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9" podUID="58b0fb12-23e0-4de8-b639-46d4045ba4c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:36:44.759886 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:44.759504    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"
Sep 17 17:36:54.758878 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:36:54.758831    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9" podUID="58b0fb12-23e0-4de8-b639-46d4045ba4c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:37:03.011890 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:03.011862    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:37:03.016090 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:03.016065    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:37:04.759499 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:04.759454    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9" podUID="58b0fb12-23e0-4de8-b639-46d4045ba4c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:37:07.665895 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:07.665865    2581 generic.go:356] "Generic (PLEG): container finished" podID="58b0fb12-23e0-4de8-b639-46d4045ba4c4" containerID="9f307c6276753b376adc542d3711ce04ef3eab7895a1aeaa4e106f773edbc57b" exitCode=0
Sep 17 17:37:07.666254 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:07.665926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9" event={"ID":"58b0fb12-23e0-4de8-b639-46d4045ba4c4","Type":"ContainerDied","Data":"9f307c6276753b376adc542d3711ce04ef3eab7895a1aeaa4e106f773edbc57b"}
Sep 17 17:37:07.727676 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:07.727655    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"
Sep 17 17:37:07.800467 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:07.800400    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjftm\" (UniqueName: \"kubernetes.io/projected/58b0fb12-23e0-4de8-b639-46d4045ba4c4-kube-api-access-tjftm\") pod \"58b0fb12-23e0-4de8-b639-46d4045ba4c4\" (UID: \"58b0fb12-23e0-4de8-b639-46d4045ba4c4\") "
Sep 17 17:37:07.800467 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:07.800463    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b0fb12-23e0-4de8-b639-46d4045ba4c4-tls-certs\") pod \"58b0fb12-23e0-4de8-b639-46d4045ba4c4\" (UID: \"58b0fb12-23e0-4de8-b639-46d4045ba4c4\") "
Sep 17 17:37:07.802397 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:07.802370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b0fb12-23e0-4de8-b639-46d4045ba4c4-kube-api-access-tjftm" (OuterVolumeSpecName: "kube-api-access-tjftm") pod "58b0fb12-23e0-4de8-b639-46d4045ba4c4" (UID: "58b0fb12-23e0-4de8-b639-46d4045ba4c4"). InnerVolumeSpecName "kube-api-access-tjftm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:07.802489 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:07.802393    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b0fb12-23e0-4de8-b639-46d4045ba4c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58b0fb12-23e0-4de8-b639-46d4045ba4c4" (UID: "58b0fb12-23e0-4de8-b639-46d4045ba4c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:07.901288 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:07.901262    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjftm\" (UniqueName: \"kubernetes.io/projected/58b0fb12-23e0-4de8-b639-46d4045ba4c4-kube-api-access-tjftm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:37:07.901288 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:07.901286    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b0fb12-23e0-4de8-b639-46d4045ba4c4-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:37:08.672266 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:08.672232    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"
Sep 17 17:37:08.672766 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:08.672233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9" event={"ID":"58b0fb12-23e0-4de8-b639-46d4045ba4c4","Type":"ContainerDied","Data":"5218f7cd36735151e0878f69ab63caa0a2f6ff8f746fdb03cc21d370ebeff589"}
Sep 17 17:37:08.672766 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:08.672380    2581 scope.go:117] "RemoveContainer" containerID="9f307c6276753b376adc542d3711ce04ef3eab7895a1aeaa4e106f773edbc57b"
Sep 17 17:37:08.695835 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:08.695813    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"]
Sep 17 17:37:08.700593 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:08.700571    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45gtmm9"]
Sep 17 17:37:08.834714 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:08.834683    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58b0fb12-23e0-4de8-b639-46d4045ba4c4" path="/var/lib/kubelet/pods/58b0fb12-23e0-4de8-b639-46d4045ba4c4/volumes"
Sep 17 17:37:19.142438 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.142357    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"]
Sep 17 17:37:19.142828 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.142735    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58b0fb12-23e0-4de8-b639-46d4045ba4c4" containerName="main"
Sep 17 17:37:19.142828 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.142748    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58b0fb12-23e0-4de8-b639-46d4045ba4c4" containerName="main"
Sep 17 17:37:19.142828 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.142765    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main"
Sep 17 17:37:19.142828 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.142772    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main"
Sep 17 17:37:19.142828 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.142797    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="storage-initializer"
Sep 17 17:37:19.142828 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.142804    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="storage-initializer"
Sep 17 17:37:19.142828 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.142810    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="llm-d-routing-sidecar"
Sep 17 17:37:19.142828 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.142815    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="llm-d-routing-sidecar"
Sep 17 17:37:19.143112 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.142872    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="llm-d-routing-sidecar"
Sep 17 17:37:19.143112 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.142881    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ddc72bf-f59e-47b9-ab06-56b3370a3f66" containerName="main"
Sep 17 17:37:19.143112 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.142890    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="58b0fb12-23e0-4de8-b639-46d4045ba4c4" containerName="main"
Sep 17 17:37:19.147432 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.147413    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.151648 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.151115    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:37:19.151648 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.151344    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 17 17:37:19.151648 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.151528    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 17 17:37:19.151648 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.151582    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 17 17:37:19.152056 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.151785    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-5tbbg\""
Sep 17 17:37:19.153111 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.153087    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 17 17:37:19.166760 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.166734    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"]
Sep 17 17:37:19.288207 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.288168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.288316 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.288235    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.288316 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.288264    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9f8s\" (UniqueName: \"kubernetes.io/projected/85ab88a8-bc06-4804-9d5c-adcee67a9e67-kube-api-access-f9f8s\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.288316 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.288310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ab88a8-bc06-4804-9d5c-adcee67a9e67-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.288439 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.288379    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.288439 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.288412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.288520 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.288467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ab88a8-bc06-4804-9d5c-adcee67a9e67-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.288520 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.288509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.389818 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.389794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.389911 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.389828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.389911 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.389856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.389911 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.389875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9f8s\" (UniqueName: \"kubernetes.io/projected/85ab88a8-bc06-4804-9d5c-adcee67a9e67-kube-api-access-f9f8s\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.390098 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.390017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ab88a8-bc06-4804-9d5c-adcee67a9e67-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.390098 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.390077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.390168 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.390113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.390168 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.390151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ab88a8-bc06-4804-9d5c-adcee67a9e67-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.390245 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.390227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.390282 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.390231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.390421 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.390403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.390505 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.390482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.390828 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.390808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ab88a8-bc06-4804-9d5c-adcee67a9e67-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.392176 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.392158    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.392617 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.392575    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ab88a8-bc06-4804-9d5c-adcee67a9e67-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.412121 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.412091    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9f8s\" (UniqueName: \"kubernetes.io/projected/85ab88a8-bc06-4804-9d5c-adcee67a9e67-kube-api-access-f9f8s\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.460246 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.460222    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:19.801747 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:19.801715    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"]
Sep 17 17:37:19.823617 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:37:19.823591    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ab88a8_bc06_4804_9d5c_adcee67a9e67.slice/crio-33649954e3ce30ef090feef6caeb910c64af4b5632d532286afdcc309611f90e WatchSource:0}: Error finding container 33649954e3ce30ef090feef6caeb910c64af4b5632d532286afdcc309611f90e: Status 404 returned error can't find the container with id 33649954e3ce30ef090feef6caeb910c64af4b5632d532286afdcc309611f90e
Sep 17 17:37:20.731181 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:20.731140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" event={"ID":"85ab88a8-bc06-4804-9d5c-adcee67a9e67","Type":"ContainerStarted","Data":"d7225532c12283317cf438e238e5c432a8214070f48edbf20439d60915f6f836"}
Sep 17 17:37:20.731669 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:20.731192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" event={"ID":"85ab88a8-bc06-4804-9d5c-adcee67a9e67","Type":"ContainerStarted","Data":"33649954e3ce30ef090feef6caeb910c64af4b5632d532286afdcc309611f90e"}
Sep 17 17:37:22.743544 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:22.743505    2581 generic.go:356] "Generic (PLEG): container finished" podID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerID="d7225532c12283317cf438e238e5c432a8214070f48edbf20439d60915f6f836" exitCode=0
Sep 17 17:37:22.743960 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:22.743586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" event={"ID":"85ab88a8-bc06-4804-9d5c-adcee67a9e67","Type":"ContainerDied","Data":"d7225532c12283317cf438e238e5c432a8214070f48edbf20439d60915f6f836"}
Sep 17 17:37:23.750150 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:23.750117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" event={"ID":"85ab88a8-bc06-4804-9d5c-adcee67a9e67","Type":"ContainerStarted","Data":"5ec92d05766fb13d548ba4abd7e922e348659c686c02c6047ba99b59b9a21293"}
Sep 17 17:37:24.460713 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:24.460680    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:29.460928 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:29.460896    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:37:29.462520 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:29.462495    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 17:37:39.461335 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:39.461294    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 17:37:49.460855 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:49.460811    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 17:37:59.460945 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:37:59.460899    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 17:38:09.461053 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:38:09.461011    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 17:38:19.461459 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:38:19.461413    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 17:38:29.461263 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:38:29.461224    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 17:38:39.460669 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:38:39.460626    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 17:38:49.460801 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:38:49.460715    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 17:38:59.460867 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:38:59.460824    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 17:39:09.470590 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:09.470555    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:39:09.478381 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:09.478352    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:39:09.502688 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:09.502638    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" podStartSLOduration=108.34124367 podStartE2EDuration="1m50.50260004s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.16135637s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:23.784107398 +0000 UTC m=+3323.207045262" watchObservedRunningTime="2026-09-17 17:39:09.50260004 +0000 UTC m=+3428.925537902"
Sep 17 17:39:13.784819 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:13.784784    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"]
Sep 17 17:39:28.806925 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:28.806850    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="main" containerID="cri-o://5ec92d05766fb13d548ba4abd7e922e348659c686c02c6047ba99b59b9a21293" gracePeriod=45
Sep 17 17:39:29.343119 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.343093    2581 generic.go:356] "Generic (PLEG): container finished" podID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerID="5ec92d05766fb13d548ba4abd7e922e348659c686c02c6047ba99b59b9a21293" exitCode=0
Sep 17 17:39:29.343322 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.343167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" event={"ID":"85ab88a8-bc06-4804-9d5c-adcee67a9e67","Type":"ContainerDied","Data":"5ec92d05766fb13d548ba4abd7e922e348659c686c02c6047ba99b59b9a21293"}
Sep 17 17:39:29.471284 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.471259    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:39:29.546019 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.545995    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9f8s\" (UniqueName: \"kubernetes.io/projected/85ab88a8-bc06-4804-9d5c-adcee67a9e67-kube-api-access-f9f8s\") pod \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") "
Sep 17 17:39:29.546156 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.546064    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-home\") pod \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") "
Sep 17 17:39:29.546156 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.546092    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-dshm\") pod \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") "
Sep 17 17:39:29.546156 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.546143    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ab88a8-bc06-4804-9d5c-adcee67a9e67-tls-certs\") pod \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") "
Sep 17 17:39:29.546330 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.546195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ab88a8-bc06-4804-9d5c-adcee67a9e67-cabundle-cert\") pod \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") "
Sep 17 17:39:29.546330 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.546251    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-model-cache\") pod \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") "
Sep 17 17:39:29.546330 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.546276    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-tmp-dir\") pod \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") "
Sep 17 17:39:29.546330 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.546313    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-kserve-provision-location\") pod \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\" (UID: \"85ab88a8-bc06-4804-9d5c-adcee67a9e67\") "
Sep 17 17:39:29.546730 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.546622    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ab88a8-bc06-4804-9d5c-adcee67a9e67-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "85ab88a8-bc06-4804-9d5c-adcee67a9e67" (UID: "85ab88a8-bc06-4804-9d5c-adcee67a9e67"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:29.546841 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.546782    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-home" (OuterVolumeSpecName: "home") pod "85ab88a8-bc06-4804-9d5c-adcee67a9e67" (UID: "85ab88a8-bc06-4804-9d5c-adcee67a9e67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:39:29.547092 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.547059    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-model-cache" (OuterVolumeSpecName: "model-cache") pod "85ab88a8-bc06-4804-9d5c-adcee67a9e67" (UID: "85ab88a8-bc06-4804-9d5c-adcee67a9e67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:39:29.548396 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.548366    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-dshm" (OuterVolumeSpecName: "dshm") pod "85ab88a8-bc06-4804-9d5c-adcee67a9e67" (UID: "85ab88a8-bc06-4804-9d5c-adcee67a9e67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:39:29.548510 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.548429    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ab88a8-bc06-4804-9d5c-adcee67a9e67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85ab88a8-bc06-4804-9d5c-adcee67a9e67" (UID: "85ab88a8-bc06-4804-9d5c-adcee67a9e67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:29.548567 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.548552    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ab88a8-bc06-4804-9d5c-adcee67a9e67-kube-api-access-f9f8s" (OuterVolumeSpecName: "kube-api-access-f9f8s") pod "85ab88a8-bc06-4804-9d5c-adcee67a9e67" (UID: "85ab88a8-bc06-4804-9d5c-adcee67a9e67"). InnerVolumeSpecName "kube-api-access-f9f8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:39:29.558128 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.558068    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85ab88a8-bc06-4804-9d5c-adcee67a9e67" (UID: "85ab88a8-bc06-4804-9d5c-adcee67a9e67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:39:29.575739 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.575717    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85ab88a8-bc06-4804-9d5c-adcee67a9e67" (UID: "85ab88a8-bc06-4804-9d5c-adcee67a9e67"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:39:29.646950 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.646923    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:39:29.646950 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.646944    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:39:29.647110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.646957    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ab88a8-bc06-4804-9d5c-adcee67a9e67-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:39:29.647110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.646991    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ab88a8-bc06-4804-9d5c-adcee67a9e67-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:39:29.647110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.647002    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:39:29.647110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.647013    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:39:29.647110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.647025    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ab88a8-bc06-4804-9d5c-adcee67a9e67-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:39:29.647110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:29.647037    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9f8s\" (UniqueName: \"kubernetes.io/projected/85ab88a8-bc06-4804-9d5c-adcee67a9e67-kube-api-access-f9f8s\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:39:30.351063 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:30.351037    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"
Sep 17 17:39:30.351473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:30.351055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs" event={"ID":"85ab88a8-bc06-4804-9d5c-adcee67a9e67","Type":"ContainerDied","Data":"33649954e3ce30ef090feef6caeb910c64af4b5632d532286afdcc309611f90e"}
Sep 17 17:39:30.351473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:30.351120    2581 scope.go:117] "RemoveContainer" containerID="5ec92d05766fb13d548ba4abd7e922e348659c686c02c6047ba99b59b9a21293"
Sep 17 17:39:30.360469 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:30.360450    2581 scope.go:117] "RemoveContainer" containerID="d7225532c12283317cf438e238e5c432a8214070f48edbf20439d60915f6f836"
Sep 17 17:39:30.377354 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:30.377331    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"]
Sep 17 17:39:30.381939 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:30.381918    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7d4b9d6dd6h42fs"]
Sep 17 17:39:30.837889 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:30.837844    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" path="/var/lib/kubelet/pods/85ab88a8-bc06-4804-9d5c-adcee67a9e67/volumes"
Sep 17 17:39:48.736338 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.736300    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 17:39:48.736883 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.736764    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="main"
Sep 17 17:39:48.736883 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.736782    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="main"
Sep 17 17:39:48.736883 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.736794    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="storage-initializer"
Sep 17 17:39:48.736883 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.736800    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="storage-initializer"
Sep 17 17:39:48.737095 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.736889    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ab88a8-bc06-4804-9d5c-adcee67a9e67" containerName="main"
Sep 17 17:39:48.739032 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.739015    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.742061 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.742027    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 17:39:48.742268 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.742097    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:39:48.742685 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.742666    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-dw9cb\""
Sep 17 17:39:48.742830 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.742671    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 17:39:48.742991 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.742959    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 17:39:48.743391 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.743375    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 17:39:48.743463 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.743390    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-pbrnk\""
Sep 17 17:39:48.760590 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.760564    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 17:39:48.795421 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.795399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.795538 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.795428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.795538 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.795453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/645b776f-76a0-4222-adca-ca28dcea3646-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.795538 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.795519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.795670 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.795555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.795670 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.795572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.795670 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.795612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/645b776f-76a0-4222-adca-ca28dcea3646-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.795670 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.795642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9rkh\" (UniqueName: \"kubernetes.io/projected/645b776f-76a0-4222-adca-ca28dcea3646-kube-api-access-m9rkh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.896475 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.896454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/645b776f-76a0-4222-adca-ca28dcea3646-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.896621 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.896496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9rkh\" (UniqueName: \"kubernetes.io/projected/645b776f-76a0-4222-adca-ca28dcea3646-kube-api-access-m9rkh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.896621 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.896539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.896621 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.896555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.896621 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.896584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/645b776f-76a0-4222-adca-ca28dcea3646-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.896847 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.896605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.896847 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.896650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.896847 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.896672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.897044 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.896931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.897044 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.897027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.897157 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.897130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.897221 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.897155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/645b776f-76a0-4222-adca-ca28dcea3646-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.897221 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.897155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.899054 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.899033    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.899445 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.899429    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/645b776f-76a0-4222-adca-ca28dcea3646-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:48.917591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:48.917568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9rkh\" (UniqueName: \"kubernetes.io/projected/645b776f-76a0-4222-adca-ca28dcea3646-kube-api-access-m9rkh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:49.050866 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:49.050809    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:49.350675 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:49.350549    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 17:39:49.355179 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:39:49.355149    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645b776f_76a0_4222_adca_ca28dcea3646.slice/crio-482027248aee3023c6d5656204ac8aa4e4c36df6ed742c127c84986f26a9eaee WatchSource:0}: Error finding container 482027248aee3023c6d5656204ac8aa4e4c36df6ed742c127c84986f26a9eaee: Status 404 returned error can't find the container with id 482027248aee3023c6d5656204ac8aa4e4c36df6ed742c127c84986f26a9eaee
Sep 17 17:39:49.452494 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:49.452454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"645b776f-76a0-4222-adca-ca28dcea3646","Type":"ContainerStarted","Data":"ba9f6cf338a24de3a24a12a256547fe0f10e05a1d053277f7768c7c08a2e96c0"}
Sep 17 17:39:49.452636 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:49.452541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"645b776f-76a0-4222-adca-ca28dcea3646","Type":"ContainerStarted","Data":"482027248aee3023c6d5656204ac8aa4e4c36df6ed742c127c84986f26a9eaee"}
Sep 17 17:39:52.470083 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:52.470050    2581 generic.go:356] "Generic (PLEG): container finished" podID="645b776f-76a0-4222-adca-ca28dcea3646" containerID="ba9f6cf338a24de3a24a12a256547fe0f10e05a1d053277f7768c7c08a2e96c0" exitCode=0
Sep 17 17:39:52.470476 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:52.470120    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"645b776f-76a0-4222-adca-ca28dcea3646","Type":"ContainerDied","Data":"ba9f6cf338a24de3a24a12a256547fe0f10e05a1d053277f7768c7c08a2e96c0"}
Sep 17 17:39:53.476809 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:53.476773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"645b776f-76a0-4222-adca-ca28dcea3646","Type":"ContainerStarted","Data":"ebd010a9d54b0ccb1b22161f4608de921a6ca592d977362e8a0ad3f50ccbf325"}
Sep 17 17:39:53.510591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:53.510531    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.357491569 podStartE2EDuration="5.51051216s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.153020591s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:53.508338336 +0000 UTC m=+3472.931276199" watchObservedRunningTime="2026-09-17 17:39:53.51051216 +0000 UTC m=+3472.933450025"
Sep 17 17:39:54.051752 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:54.051723    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:59.051070 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:59.051035    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:39:59.052226 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:39:59.052199    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 17:40:09.051850 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:09.051806    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 17:40:19.051747 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:19.051657    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 17:40:29.051959 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:29.051914    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 17:40:39.051784 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:39.051745    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 17:40:49.052113 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:49.052076    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 17:40:52.898844 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:52.898810    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"]
Sep 17 17:40:52.902353 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:52.902335    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"
Sep 17 17:40:52.904952 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:52.904913    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 17:40:52.905087 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:52.905067    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 17:40:52.905171 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:52.905153    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 17 17:40:52.905786 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:52.905767    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-fg7b7\""
Sep 17 17:40:52.915656 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:52.915615    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"]
Sep 17 17:40:53.026855 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:53.026817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7md5v\" (UniqueName: \"kubernetes.io/projected/4b977b1f-b03a-499d-9248-8a60290148f2-kube-api-access-7md5v\") pod \"e2e-pvc-lora-download-ghdss\" (UID: \"4b977b1f-b03a-499d-9248-8a60290148f2\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"
Sep 17 17:40:53.127935 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:53.127904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7md5v\" (UniqueName: \"kubernetes.io/projected/4b977b1f-b03a-499d-9248-8a60290148f2-kube-api-access-7md5v\") pod \"e2e-pvc-lora-download-ghdss\" (UID: \"4b977b1f-b03a-499d-9248-8a60290148f2\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"
Sep 17 17:40:53.137647 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:53.137616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7md5v\" (UniqueName: \"kubernetes.io/projected/4b977b1f-b03a-499d-9248-8a60290148f2-kube-api-access-7md5v\") pod \"e2e-pvc-lora-download-ghdss\" (UID: \"4b977b1f-b03a-499d-9248-8a60290148f2\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"
Sep 17 17:40:57.563960 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:57.563925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"e2e-pvc-lora-download-ghdss\" (UID: \"4b977b1f-b03a-499d-9248-8a60290148f2\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"
Sep 17 17:40:57.570695 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:57.570669    2581 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"e2e-pvc-lora-download-ghdss\" (UID: \"4b977b1f-b03a-499d-9248-8a60290148f2\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"
Sep 17 17:40:57.664521 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:57.664492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"e2e-pvc-lora-download-ghdss\" (UID: \"4b977b1f-b03a-499d-9248-8a60290148f2\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"
Sep 17 17:40:57.664655 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:57.664634    2581 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"e2e-pvc-lora-download-ghdss\" (UID: \"4b977b1f-b03a-499d-9248-8a60290148f2\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"
Sep 17 17:40:57.668672 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:57.668654    2581 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"e2e-pvc-lora-download-ghdss\" (UID: \"4b977b1f-b03a-499d-9248-8a60290148f2\") DevicePath \"csi-e0bbabeed69b72bdb1fda780b761736cff5ee1ee63f932ffc28a84d2a27c3dbe\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"
Sep 17 17:40:57.891608 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:57.891567    2581 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"e2e-pvc-lora-download-ghdss\" (UID: \"4b977b1f-b03a-499d-9248-8a60290148f2\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/76b44bf1d5f98fa88e8e38d0816d624e6412f2416e862469cfe9db7b87b3f5e4/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"
Sep 17 17:40:57.905676 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:57.905650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"e2e-pvc-lora-download-ghdss\" (UID: \"4b977b1f-b03a-499d-9248-8a60290148f2\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"
Sep 17 17:40:58.017931 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:58.017891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"
Sep 17 17:40:58.295950 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:58.295922    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"]
Sep 17 17:40:58.297881 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:40:58.297851    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b977b1f_b03a_499d_9248_8a60290148f2.slice/crio-5289b32c07fb6c3197ee670771eb00dbb11e6ee2f25416c35b9338e73750032c WatchSource:0}: Error finding container 5289b32c07fb6c3197ee670771eb00dbb11e6ee2f25416c35b9338e73750032c: Status 404 returned error can't find the container with id 5289b32c07fb6c3197ee670771eb00dbb11e6ee2f25416c35b9338e73750032c
Sep 17 17:40:58.791110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:58.791066    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss" event={"ID":"4b977b1f-b03a-499d-9248-8a60290148f2","Type":"ContainerStarted","Data":"b179b06b1edf18f2c80b9a3a806cd10319795ba8ff4c6e3c38e3f7db19f12799"}
Sep 17 17:40:58.791110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:58.791116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss" event={"ID":"4b977b1f-b03a-499d-9248-8a60290148f2","Type":"ContainerStarted","Data":"5289b32c07fb6c3197ee670771eb00dbb11e6ee2f25416c35b9338e73750032c"}
Sep 17 17:40:58.810975 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:58.810913    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss" podStartSLOduration=9.810899402 podStartE2EDuration="9.810899402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:58.80857559 +0000 UTC m=+3538.231513449" watchObservedRunningTime="2026-09-17 17:40:58.810899402 +0000 UTC m=+3538.233837319"
Sep 17 17:40:59.052154 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:40:59.052054    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 17:41:00.804202 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:00.804166    2581 generic.go:356] "Generic (PLEG): container finished" podID="4b977b1f-b03a-499d-9248-8a60290148f2" containerID="b179b06b1edf18f2c80b9a3a806cd10319795ba8ff4c6e3c38e3f7db19f12799" exitCode=0
Sep 17 17:41:00.804591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:00.804216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss" event={"ID":"4b977b1f-b03a-499d-9248-8a60290148f2","Type":"ContainerDied","Data":"b179b06b1edf18f2c80b9a3a806cd10319795ba8ff4c6e3c38e3f7db19f12799"}
Sep 17 17:41:01.948458 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:01.948435    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"
Sep 17 17:41:02.103583 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:02.103493    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") pod \"4b977b1f-b03a-499d-9248-8a60290148f2\" (UID: \"4b977b1f-b03a-499d-9248-8a60290148f2\") "
Sep 17 17:41:02.103740 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:02.103591    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7md5v\" (UniqueName: \"kubernetes.io/projected/4b977b1f-b03a-499d-9248-8a60290148f2-kube-api-access-7md5v\") pod \"4b977b1f-b03a-499d-9248-8a60290148f2\" (UID: \"4b977b1f-b03a-499d-9248-8a60290148f2\") "
Sep 17 17:41:02.105864 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:02.105838    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b977b1f-b03a-499d-9248-8a60290148f2-kube-api-access-7md5v" (OuterVolumeSpecName: "kube-api-access-7md5v") pod "4b977b1f-b03a-499d-9248-8a60290148f2" (UID: "4b977b1f-b03a-499d-9248-8a60290148f2"). InnerVolumeSpecName "kube-api-access-7md5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:41:02.107553 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:02.107529    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505" (OuterVolumeSpecName: "model-storage") pod "4b977b1f-b03a-499d-9248-8a60290148f2" (UID: "4b977b1f-b03a-499d-9248-8a60290148f2"). InnerVolumeSpecName "pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 17:41:02.204715 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:02.204677    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7md5v\" (UniqueName: \"kubernetes.io/projected/4b977b1f-b03a-499d-9248-8a60290148f2-kube-api-access-7md5v\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:41:02.204848 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:02.204745    2581 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") on node \"ip-10-0-140-49.ec2.internal\" "
Sep 17 17:41:02.815269 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:02.815236    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"
Sep 17 17:41:02.815269 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:02.815250    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss" event={"ID":"4b977b1f-b03a-499d-9248-8a60290148f2","Type":"ContainerDied","Data":"5289b32c07fb6c3197ee670771eb00dbb11e6ee2f25416c35b9338e73750032c"}
Sep 17 17:41:02.815484 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:02.815288    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5289b32c07fb6c3197ee670771eb00dbb11e6ee2f25416c35b9338e73750032c"
Sep 17 17:41:03.173310 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:03.173290    2581 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505") on node "ip-10-0-140-49.ec2.internal"
Sep 17 17:41:03.213488 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:03.213463    2581 reconciler_common.go:299] "Volume detached for volume \"pvc-ee8a953e-381f-4eaf-9d9a-b1637e28015e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0200295866b2d3505\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"csi-e0bbabeed69b72bdb1fda780b761736cff5ee1ee63f932ffc28a84d2a27c3dbe\""
Sep 17 17:41:09.053412 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:09.053365    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 17:41:19.051239 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:19.051189    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 17:41:29.061591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:29.061557    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:41:29.069489 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:29.069463    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:41:44.250975 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:44.250887    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 17:41:59.273035 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:59.272993    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="main" containerID="cri-o://ebd010a9d54b0ccb1b22161f4608de921a6ca592d977362e8a0ad3f50ccbf325" gracePeriod=45
Sep 17 17:41:59.930094 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:41:59.930065    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:42:00.065482 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.065403    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-home\") pod \"645b776f-76a0-4222-adca-ca28dcea3646\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") "
Sep 17 17:42:00.065631 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.065482    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-tmp-dir\") pod \"645b776f-76a0-4222-adca-ca28dcea3646\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") "
Sep 17 17:42:00.065631 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.065541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-kserve-provision-location\") pod \"645b776f-76a0-4222-adca-ca28dcea3646\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") "
Sep 17 17:42:00.065631 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.065582    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/645b776f-76a0-4222-adca-ca28dcea3646-tls-certs\") pod \"645b776f-76a0-4222-adca-ca28dcea3646\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") "
Sep 17 17:42:00.065631 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.065614    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-model-cache\") pod \"645b776f-76a0-4222-adca-ca28dcea3646\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") "
Sep 17 17:42:00.065865 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.065643    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-dshm\") pod \"645b776f-76a0-4222-adca-ca28dcea3646\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") "
Sep 17 17:42:00.065865 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.065698    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9rkh\" (UniqueName: \"kubernetes.io/projected/645b776f-76a0-4222-adca-ca28dcea3646-kube-api-access-m9rkh\") pod \"645b776f-76a0-4222-adca-ca28dcea3646\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") "
Sep 17 17:42:00.065865 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.065765    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/645b776f-76a0-4222-adca-ca28dcea3646-cabundle-cert\") pod \"645b776f-76a0-4222-adca-ca28dcea3646\" (UID: \"645b776f-76a0-4222-adca-ca28dcea3646\") "
Sep 17 17:42:00.066063 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.065913    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-model-cache" (OuterVolumeSpecName: "model-cache") pod "645b776f-76a0-4222-adca-ca28dcea3646" (UID: "645b776f-76a0-4222-adca-ca28dcea3646"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:00.066124 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.066058    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:42:00.066179 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.066121    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-home" (OuterVolumeSpecName: "home") pod "645b776f-76a0-4222-adca-ca28dcea3646" (UID: "645b776f-76a0-4222-adca-ca28dcea3646"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:00.066481 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.066451    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645b776f-76a0-4222-adca-ca28dcea3646-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "645b776f-76a0-4222-adca-ca28dcea3646" (UID: "645b776f-76a0-4222-adca-ca28dcea3646"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:00.068334 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.068310    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645b776f-76a0-4222-adca-ca28dcea3646-kube-api-access-m9rkh" (OuterVolumeSpecName: "kube-api-access-m9rkh") pod "645b776f-76a0-4222-adca-ca28dcea3646" (UID: "645b776f-76a0-4222-adca-ca28dcea3646"). InnerVolumeSpecName "kube-api-access-m9rkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:00.069355 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.069326    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-dshm" (OuterVolumeSpecName: "dshm") pod "645b776f-76a0-4222-adca-ca28dcea3646" (UID: "645b776f-76a0-4222-adca-ca28dcea3646"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:00.069355 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.069334    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/645b776f-76a0-4222-adca-ca28dcea3646-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "645b776f-76a0-4222-adca-ca28dcea3646" (UID: "645b776f-76a0-4222-adca-ca28dcea3646"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:00.077093 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.077067    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "645b776f-76a0-4222-adca-ca28dcea3646" (UID: "645b776f-76a0-4222-adca-ca28dcea3646"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:00.093947 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.093922    2581 generic.go:356] "Generic (PLEG): container finished" podID="645b776f-76a0-4222-adca-ca28dcea3646" containerID="ebd010a9d54b0ccb1b22161f4608de921a6ca592d977362e8a0ad3f50ccbf325" exitCode=0
Sep 17 17:42:00.094062 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.094001    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 17:42:00.094131 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.093996    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"645b776f-76a0-4222-adca-ca28dcea3646","Type":"ContainerDied","Data":"ebd010a9d54b0ccb1b22161f4608de921a6ca592d977362e8a0ad3f50ccbf325"}
Sep 17 17:42:00.094131 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.094127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"645b776f-76a0-4222-adca-ca28dcea3646","Type":"ContainerDied","Data":"482027248aee3023c6d5656204ac8aa4e4c36df6ed742c127c84986f26a9eaee"}
Sep 17 17:42:00.094233 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.094150    2581 scope.go:117] "RemoveContainer" containerID="ebd010a9d54b0ccb1b22161f4608de921a6ca592d977362e8a0ad3f50ccbf325"
Sep 17 17:42:00.095184 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.095144    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "645b776f-76a0-4222-adca-ca28dcea3646" (UID: "645b776f-76a0-4222-adca-ca28dcea3646"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:00.102407 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.102365    2581 scope.go:117] "RemoveContainer" containerID="ba9f6cf338a24de3a24a12a256547fe0f10e05a1d053277f7768c7c08a2e96c0"
Sep 17 17:42:00.109909 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.109881    2581 scope.go:117] "RemoveContainer" containerID="ebd010a9d54b0ccb1b22161f4608de921a6ca592d977362e8a0ad3f50ccbf325"
Sep 17 17:42:00.110235 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:42:00.110217    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd010a9d54b0ccb1b22161f4608de921a6ca592d977362e8a0ad3f50ccbf325\": container with ID starting with ebd010a9d54b0ccb1b22161f4608de921a6ca592d977362e8a0ad3f50ccbf325 not found: ID does not exist" containerID="ebd010a9d54b0ccb1b22161f4608de921a6ca592d977362e8a0ad3f50ccbf325"
Sep 17 17:42:00.110287 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.110242    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd010a9d54b0ccb1b22161f4608de921a6ca592d977362e8a0ad3f50ccbf325"} err="failed to get container status \"ebd010a9d54b0ccb1b22161f4608de921a6ca592d977362e8a0ad3f50ccbf325\": rpc error: code = NotFound desc = could not find container \"ebd010a9d54b0ccb1b22161f4608de921a6ca592d977362e8a0ad3f50ccbf325\": container with ID starting with ebd010a9d54b0ccb1b22161f4608de921a6ca592d977362e8a0ad3f50ccbf325 not found: ID does not exist"
Sep 17 17:42:00.110287 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.110258    2581 scope.go:117] "RemoveContainer" containerID="ba9f6cf338a24de3a24a12a256547fe0f10e05a1d053277f7768c7c08a2e96c0"
Sep 17 17:42:00.110467 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:42:00.110451    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9f6cf338a24de3a24a12a256547fe0f10e05a1d053277f7768c7c08a2e96c0\": container with ID starting with ba9f6cf338a24de3a24a12a256547fe0f10e05a1d053277f7768c7c08a2e96c0 not found: ID does not exist" containerID="ba9f6cf338a24de3a24a12a256547fe0f10e05a1d053277f7768c7c08a2e96c0"
Sep 17 17:42:00.110504 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.110474    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9f6cf338a24de3a24a12a256547fe0f10e05a1d053277f7768c7c08a2e96c0"} err="failed to get container status \"ba9f6cf338a24de3a24a12a256547fe0f10e05a1d053277f7768c7c08a2e96c0\": rpc error: code = NotFound desc = could not find container \"ba9f6cf338a24de3a24a12a256547fe0f10e05a1d053277f7768c7c08a2e96c0\": container with ID starting with ba9f6cf338a24de3a24a12a256547fe0f10e05a1d053277f7768c7c08a2e96c0 not found: ID does not exist"
Sep 17 17:42:00.166856 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.166833    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:42:00.166856 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.166854    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9rkh\" (UniqueName: \"kubernetes.io/projected/645b776f-76a0-4222-adca-ca28dcea3646-kube-api-access-m9rkh\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:42:00.166997 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.166866    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/645b776f-76a0-4222-adca-ca28dcea3646-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:42:00.166997 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.166875    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:42:00.166997 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.166883    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:42:00.166997 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.166891    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/645b776f-76a0-4222-adca-ca28dcea3646-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:42:00.166997 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.166899    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/645b776f-76a0-4222-adca-ca28dcea3646-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:42:00.421269 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.421237    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 17:42:00.423761 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.423737    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 17:42:00.835571 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:00.835537    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="645b776f-76a0-4222-adca-ca28dcea3646" path="/var/lib/kubelet/pods/645b776f-76a0-4222-adca-ca28dcea3646/volumes"
Sep 17 17:42:03.040501 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:03.040469    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:42:03.045709 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:03.045689    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:42:45.275057 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.274948    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"]
Sep 17 17:42:45.275691 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.275352    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="main"
Sep 17 17:42:45.275691 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.275364    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="main"
Sep 17 17:42:45.275691 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.275381    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b977b1f-b03a-499d-9248-8a60290148f2" containerName="storage-initializer"
Sep 17 17:42:45.275691 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.275387    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b977b1f-b03a-499d-9248-8a60290148f2" containerName="storage-initializer"
Sep 17 17:42:45.275691 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.275399    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="storage-initializer"
Sep 17 17:42:45.275691 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.275406    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="storage-initializer"
Sep 17 17:42:45.275691 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.275466    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="645b776f-76a0-4222-adca-ca28dcea3646" containerName="main"
Sep 17 17:42:45.275691 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.275473    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b977b1f-b03a-499d-9248-8a60290148f2" containerName="storage-initializer"
Sep 17 17:42:45.278491 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.278468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.281757 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.281731    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 17:42:45.281887 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.281780    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 17:42:45.281887 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.281781    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-d4m52\""
Sep 17 17:42:45.282027 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.281745    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 17:42:45.290285 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.290263    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"]
Sep 17 17:42:45.310565 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.310532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.310565 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.310566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.310770 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.310639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16e3353c-5081-4fa7-a36e-14cf10d1ab47-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.310770 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.310666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmvf\" (UniqueName: \"kubernetes.io/projected/16e3353c-5081-4fa7-a36e-14cf10d1ab47-kube-api-access-xnmvf\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.310770 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.310693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.310921 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.310796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.411598 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.411552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.411598 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.411603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.411868 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.411621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.411868 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.411655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16e3353c-5081-4fa7-a36e-14cf10d1ab47-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.411868 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.411687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmvf\" (UniqueName: \"kubernetes.io/projected/16e3353c-5081-4fa7-a36e-14cf10d1ab47-kube-api-access-xnmvf\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.411868 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.411716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.412118 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.412060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.412118 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.412101    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.412223 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.412142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.414040 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.414017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.414298 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.414281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16e3353c-5081-4fa7-a36e-14cf10d1ab47-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.420514 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.420480    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmvf\" (UniqueName: \"kubernetes.io/projected/16e3353c-5081-4fa7-a36e-14cf10d1ab47-kube-api-access-xnmvf\") pod \"scheduler-inline-config-test-kserve-8655bc757-52jcv\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.590110 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.590026    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:45.867088 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:45.867059    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"]
Sep 17 17:42:45.869754 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:42:45.869723    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e3353c_5081_4fa7_a36e_14cf10d1ab47.slice/crio-18cfc11f9ff5d66b7bebc9cf23e29db2cdcc1301ab458870b2a34a3fcfd5a5a9 WatchSource:0}: Error finding container 18cfc11f9ff5d66b7bebc9cf23e29db2cdcc1301ab458870b2a34a3fcfd5a5a9: Status 404 returned error can't find the container with id 18cfc11f9ff5d66b7bebc9cf23e29db2cdcc1301ab458870b2a34a3fcfd5a5a9
Sep 17 17:42:46.308593 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:46.308555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv" event={"ID":"16e3353c-5081-4fa7-a36e-14cf10d1ab47","Type":"ContainerStarted","Data":"0f252048092e01a525dd035a79a6811d2b26691071bbc6886160bbe68d4cc0f5"}
Sep 17 17:42:46.309039 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:46.308609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv" event={"ID":"16e3353c-5081-4fa7-a36e-14cf10d1ab47","Type":"ContainerStarted","Data":"18cfc11f9ff5d66b7bebc9cf23e29db2cdcc1301ab458870b2a34a3fcfd5a5a9"}
Sep 17 17:42:46.345804 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:46.345746    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv" podStartSLOduration=1.345726601 podStartE2EDuration="1.345726601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:46.345399247 +0000 UTC m=+3645.768337111" watchObservedRunningTime="2026-09-17 17:42:46.345726601 +0000 UTC m=+3645.768664469"
Sep 17 17:42:46.591169 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:46.591092    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:55.590802 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:55.590760    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:55.602935 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:55.602910    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:42:56.364069 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:42:56.364033    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:43:20.248996 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.248884    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"]
Sep 17 17:43:20.258181 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:43:20.258145    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:43:20.258181 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:43:20.258181 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:43:20.258181 ip-10-0-140-49 kubenswrapper[2581]:  > containerID="0f252048092e01a525dd035a79a6811d2b26691071bbc6886160bbe68d4cc0f5" cmd=["/bin/sleep","15"]
Sep 17 17:43:20.258419 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.258194    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv" message="" err=<
Sep 17 17:43:20.258419 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:43:20.258419 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:43:20.258419 ip-10-0-140-49 kubenswrapper[2581]:  >
Sep 17 17:43:20.258419 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:43:20.258206    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:43:20.258419 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:43:20.258419 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:43:20.258419 ip-10-0-140-49 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv" podUID="16e3353c-5081-4fa7-a36e-14cf10d1ab47" containerName="main" containerID="cri-o://0f252048092e01a525dd035a79a6811d2b26691071bbc6886160bbe68d4cc0f5"
Sep 17 17:43:20.258419 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.258240    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv" podUID="16e3353c-5081-4fa7-a36e-14cf10d1ab47" containerName="main" containerID="cri-o://0f252048092e01a525dd035a79a6811d2b26691071bbc6886160bbe68d4cc0f5" gracePeriod=60
Sep 17 17:43:20.354272 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.354229    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv" podUID="16e3353c-5081-4fa7-a36e-14cf10d1ab47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 17:43:20.476743 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.476712    2581 generic.go:356] "Generic (PLEG): container finished" podID="16e3353c-5081-4fa7-a36e-14cf10d1ab47" containerID="0f252048092e01a525dd035a79a6811d2b26691071bbc6886160bbe68d4cc0f5" exitCode=0
Sep 17 17:43:20.476868 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.476784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv" event={"ID":"16e3353c-5081-4fa7-a36e-14cf10d1ab47","Type":"ContainerDied","Data":"0f252048092e01a525dd035a79a6811d2b26691071bbc6886160bbe68d4cc0f5"}
Sep 17 17:43:20.606136 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.606112    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:43:20.608136 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.608115    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-model-cache\") pod \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") "
Sep 17 17:43:20.608192 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.608144    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-dshm\") pod \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") "
Sep 17 17:43:20.608192 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.608167    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnmvf\" (UniqueName: \"kubernetes.io/projected/16e3353c-5081-4fa7-a36e-14cf10d1ab47-kube-api-access-xnmvf\") pod \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") "
Sep 17 17:43:20.608192 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.608184    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-home\") pod \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") "
Sep 17 17:43:20.608337 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.608236    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16e3353c-5081-4fa7-a36e-14cf10d1ab47-tls-certs\") pod \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") "
Sep 17 17:43:20.608337 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.608260    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-tmp-dir\") pod \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\" (UID: \"16e3353c-5081-4fa7-a36e-14cf10d1ab47\") "
Sep 17 17:43:20.608444 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.608402    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-model-cache" (OuterVolumeSpecName: "model-cache") pod "16e3353c-5081-4fa7-a36e-14cf10d1ab47" (UID: "16e3353c-5081-4fa7-a36e-14cf10d1ab47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:43:20.608507 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.608489    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-home" (OuterVolumeSpecName: "home") pod "16e3353c-5081-4fa7-a36e-14cf10d1ab47" (UID: "16e3353c-5081-4fa7-a36e-14cf10d1ab47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:43:20.608610 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.608594    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:43:20.608657 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.608617    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:43:20.608657 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.608615    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16e3353c-5081-4fa7-a36e-14cf10d1ab47" (UID: "16e3353c-5081-4fa7-a36e-14cf10d1ab47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:43:20.610253 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.610233    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-dshm" (OuterVolumeSpecName: "dshm") pod "16e3353c-5081-4fa7-a36e-14cf10d1ab47" (UID: "16e3353c-5081-4fa7-a36e-14cf10d1ab47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:43:20.611335 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.611304    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e3353c-5081-4fa7-a36e-14cf10d1ab47-kube-api-access-xnmvf" (OuterVolumeSpecName: "kube-api-access-xnmvf") pod "16e3353c-5081-4fa7-a36e-14cf10d1ab47" (UID: "16e3353c-5081-4fa7-a36e-14cf10d1ab47"). InnerVolumeSpecName "kube-api-access-xnmvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:43:20.611451 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.611340    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e3353c-5081-4fa7-a36e-14cf10d1ab47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16e3353c-5081-4fa7-a36e-14cf10d1ab47" (UID: "16e3353c-5081-4fa7-a36e-14cf10d1ab47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:20.709358 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.709329    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16e3353c-5081-4fa7-a36e-14cf10d1ab47-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:43:20.709358 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.709354    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:43:20.709358 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.709362    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16e3353c-5081-4fa7-a36e-14cf10d1ab47-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:43:20.709559 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:20.709370    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnmvf\" (UniqueName: \"kubernetes.io/projected/16e3353c-5081-4fa7-a36e-14cf10d1ab47-kube-api-access-xnmvf\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:43:21.483162 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:21.483129    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"
Sep 17 17:43:21.483162 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:21.483155    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv" event={"ID":"16e3353c-5081-4fa7-a36e-14cf10d1ab47","Type":"ContainerDied","Data":"18cfc11f9ff5d66b7bebc9cf23e29db2cdcc1301ab458870b2a34a3fcfd5a5a9"}
Sep 17 17:43:21.483697 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:21.483215    2581 scope.go:117] "RemoveContainer" containerID="0f252048092e01a525dd035a79a6811d2b26691071bbc6886160bbe68d4cc0f5"
Sep 17 17:43:21.524342 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:21.524301    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"]
Sep 17 17:43:21.532525 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:21.532497    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-52jcv"]
Sep 17 17:43:22.836519 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:22.836486    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e3353c-5081-4fa7-a36e-14cf10d1ab47" path="/var/lib/kubelet/pods/16e3353c-5081-4fa7-a36e-14cf10d1ab47/volumes"
Sep 17 17:43:54.595454 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:54.595412    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"]
Sep 17 17:43:54.603929 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:54.603882    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ghdss"]
Sep 17 17:43:54.835745 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:54.835698    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b977b1f-b03a-499d-9248-8a60290148f2" path="/var/lib/kubelet/pods/4b977b1f-b03a-499d-9248-8a60290148f2/volumes"
Sep 17 17:43:56.712788 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.712747    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"]
Sep 17 17:43:56.713238 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.713128    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e3353c-5081-4fa7-a36e-14cf10d1ab47" containerName="main"
Sep 17 17:43:56.713238 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.713144    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e3353c-5081-4fa7-a36e-14cf10d1ab47" containerName="main"
Sep 17 17:43:56.713238 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.713209    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e3353c-5081-4fa7-a36e-14cf10d1ab47" containerName="main"
Sep 17 17:43:56.716138 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.716117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.718658 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.718626    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:43:56.718658 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.718655    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-l7xw4\""
Sep 17 17:43:56.718873 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.718631    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 17:43:56.719125 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.719106    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 17:43:56.719248 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.719228    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 17:43:56.719348 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.719328    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 17 17:43:56.732016 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.731990    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"]
Sep 17 17:43:56.794035 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.794008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c956d0ea-2a22-47c6-af38-2bcc521d2746-cabundle-cert\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.794150 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.794038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-dshm\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.794150 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.794063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c956d0ea-2a22-47c6-af38-2bcc521d2746-tls-certs\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.794150 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.794141    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-tmp-dir\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.794271 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.794184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-home\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.794271 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.794206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-model-cache\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.794271 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.794225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7lxn\" (UniqueName: \"kubernetes.io/projected/c956d0ea-2a22-47c6-af38-2bcc521d2746-kube-api-access-n7lxn\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.794369 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.794278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-kserve-provision-location\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.894960 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.894932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c956d0ea-2a22-47c6-af38-2bcc521d2746-tls-certs\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.895102 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.894991    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-tmp-dir\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.895102 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.895024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-home\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.895102 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.895046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-model-cache\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.895102 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.895065    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7lxn\" (UniqueName: \"kubernetes.io/projected/c956d0ea-2a22-47c6-af38-2bcc521d2746-kube-api-access-n7lxn\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.895308 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.895113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-kserve-provision-location\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.895308 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.895148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c956d0ea-2a22-47c6-af38-2bcc521d2746-cabundle-cert\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.895308 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.895171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-dshm\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.895682 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.895649    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-model-cache\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.895821 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.895682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-kserve-provision-location\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.895821 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.895741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-tmp-dir\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.895821 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.895798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-home\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.896093 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.896070    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c956d0ea-2a22-47c6-af38-2bcc521d2746-cabundle-cert\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.897638 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.897611    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-dshm\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.897844 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.897820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c956d0ea-2a22-47c6-af38-2bcc521d2746-tls-certs\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:56.906222 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:56.906196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7lxn\" (UniqueName: \"kubernetes.io/projected/c956d0ea-2a22-47c6-af38-2bcc521d2746-kube-api-access-n7lxn\") pod \"tls-verification-test-kserve-d8d857656-5c5gl\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:57.028540 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:57.028474    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:43:57.330683 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:57.330653    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"]
Sep 17 17:43:57.332017 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:43:57.331985    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc956d0ea_2a22_47c6_af38_2bcc521d2746.slice/crio-c1cfa753aa04f517bbe47ed6b760cad6aa1c9cedd8dfc7623b4ac717044c98d8 WatchSource:0}: Error finding container c1cfa753aa04f517bbe47ed6b760cad6aa1c9cedd8dfc7623b4ac717044c98d8: Status 404 returned error can't find the container with id c1cfa753aa04f517bbe47ed6b760cad6aa1c9cedd8dfc7623b4ac717044c98d8
Sep 17 17:43:57.673869 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:57.673776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" event={"ID":"c956d0ea-2a22-47c6-af38-2bcc521d2746","Type":"ContainerStarted","Data":"0886ac8db7d3889a2c5758fe6f0e041303d59d0d498f8d0ecea4e6f52f849315"}
Sep 17 17:43:57.674068 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:57.673870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" event={"ID":"c956d0ea-2a22-47c6-af38-2bcc521d2746","Type":"ContainerStarted","Data":"c1cfa753aa04f517bbe47ed6b760cad6aa1c9cedd8dfc7623b4ac717044c98d8"}
Sep 17 17:43:59.684830 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:59.684801    2581 generic.go:356] "Generic (PLEG): container finished" podID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerID="0886ac8db7d3889a2c5758fe6f0e041303d59d0d498f8d0ecea4e6f52f849315" exitCode=0
Sep 17 17:43:59.685262 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:43:59.684847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" event={"ID":"c956d0ea-2a22-47c6-af38-2bcc521d2746","Type":"ContainerDied","Data":"0886ac8db7d3889a2c5758fe6f0e041303d59d0d498f8d0ecea4e6f52f849315"}
Sep 17 17:44:00.694372 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:00.694330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" event={"ID":"c956d0ea-2a22-47c6-af38-2bcc521d2746","Type":"ContainerStarted","Data":"e7fdff9e0d1fa49ec5111add4a29c109ad0c4250f29d8edd0d3d953f4bc3541e"}
Sep 17 17:44:00.725088 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:00.725040    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" podStartSLOduration=2.518011063 podStartE2EDuration="4.725026113s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.20701505s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:00.723872805 +0000 UTC m=+3720.146810667" watchObservedRunningTime="2026-09-17 17:44:00.725026113 +0000 UTC m=+3720.147963975"
Sep 17 17:44:01.029473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:01.029383    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:44:07.028852 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:07.028815    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:44:07.029920 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:07.029885    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 17 17:44:17.029076 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:17.029031    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 17 17:44:25.938198 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:25.938162    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"]
Sep 17 17:44:25.942105 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:25.942085    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"
Sep 17 17:44:25.944476 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:25.944451    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 17:44:25.945360 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:25.945333    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-nlfkl\""
Sep 17 17:44:25.945469 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:25.945389    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 17:44:25.945469 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:25.945430    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 17:44:25.945578 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:25.945559    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-px72s\""
Sep 17 17:44:25.953427 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:25.953406    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"]
Sep 17 17:44:26.028759 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:26.028730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84fe36a3-293a-4285-aaf5-00ee54c8f157-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn\" (UID: \"84fe36a3-293a-4285-aaf5-00ee54c8f157\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"
Sep 17 17:44:26.028877 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:26.028771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwbl5\" (UniqueName: \"kubernetes.io/projected/84fe36a3-293a-4285-aaf5-00ee54c8f157-kube-api-access-nwbl5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn\" (UID: \"84fe36a3-293a-4285-aaf5-00ee54c8f157\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"
Sep 17 17:44:26.129521 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:26.129489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84fe36a3-293a-4285-aaf5-00ee54c8f157-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn\" (UID: \"84fe36a3-293a-4285-aaf5-00ee54c8f157\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"
Sep 17 17:44:26.129637 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:26.129535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwbl5\" (UniqueName: \"kubernetes.io/projected/84fe36a3-293a-4285-aaf5-00ee54c8f157-kube-api-access-nwbl5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn\" (UID: \"84fe36a3-293a-4285-aaf5-00ee54c8f157\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"
Sep 17 17:44:26.131949 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:26.131931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84fe36a3-293a-4285-aaf5-00ee54c8f157-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn\" (UID: \"84fe36a3-293a-4285-aaf5-00ee54c8f157\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"
Sep 17 17:44:26.139147 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:26.139128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwbl5\" (UniqueName: \"kubernetes.io/projected/84fe36a3-293a-4285-aaf5-00ee54c8f157-kube-api-access-nwbl5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn\" (UID: \"84fe36a3-293a-4285-aaf5-00ee54c8f157\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"
Sep 17 17:44:26.252760 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:26.252687    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"
Sep 17 17:44:26.543951 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:26.543918    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"]
Sep 17 17:44:26.545495 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:44:26.545468    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fe36a3_293a_4285_aaf5_00ee54c8f157.slice/crio-8ef4d72ff184506e4ccf557fab30cecc5c5e9139d01e531b10c0fe77e6024142 WatchSource:0}: Error finding container 8ef4d72ff184506e4ccf557fab30cecc5c5e9139d01e531b10c0fe77e6024142: Status 404 returned error can't find the container with id 8ef4d72ff184506e4ccf557fab30cecc5c5e9139d01e531b10c0fe77e6024142
Sep 17 17:44:26.837247 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:26.837161    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn" event={"ID":"84fe36a3-293a-4285-aaf5-00ee54c8f157","Type":"ContainerStarted","Data":"0b0007704f172717beb39dba4f892f3ce9b8488923d0729a1033a2a24a75dcf9"}
Sep 17 17:44:26.837247 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:26.837215    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"
Sep 17 17:44:26.837247 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:26.837237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn" event={"ID":"84fe36a3-293a-4285-aaf5-00ee54c8f157","Type":"ContainerStarted","Data":"8ef4d72ff184506e4ccf557fab30cecc5c5e9139d01e531b10c0fe77e6024142"}
Sep 17 17:44:26.859733 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:26.859685    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn" podStartSLOduration=1.8596711849999998 podStartE2EDuration="1.859671185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:26.858056848 +0000 UTC m=+3746.280994711" watchObservedRunningTime="2026-09-17 17:44:26.859671185 +0000 UTC m=+3746.282609047"
Sep 17 17:44:27.028978 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:27.028933    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 17 17:44:37.028812 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:37.028762    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 17 17:44:47.029567 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:47.029478    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 17 17:44:57.028832 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:57.028784    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 17 17:44:57.844430 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:44:57.844400    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"
Sep 17 17:45:00.215073 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:00.215038    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"]
Sep 17 17:45:07.029026 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:07.028959    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 17 17:45:07.844088 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:07.844039    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn" podUID="84fe36a3-293a-4285-aaf5-00ee54c8f157" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:45:15.236485 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:15.236420    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn" podUID="84fe36a3-293a-4285-aaf5-00ee54c8f157" containerName="main" containerID="cri-o://0b0007704f172717beb39dba4f892f3ce9b8488923d0729a1033a2a24a75dcf9" gracePeriod=45
Sep 17 17:45:17.028988 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:17.028933    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 17 17:45:17.843933 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:17.843895    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn" podUID="84fe36a3-293a-4285-aaf5-00ee54c8f157" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:45:27.029686 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:27.029638    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 17 17:45:27.844216 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:27.844176    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn" podUID="84fe36a3-293a-4285-aaf5-00ee54c8f157" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:45:27.844404 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:27.844286    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"
Sep 17 17:45:37.028986 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:37.028912    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 17 17:45:37.843810 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:37.843770    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn" podUID="84fe36a3-293a-4285-aaf5-00ee54c8f157" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:45:45.487903 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:45.487877    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"
Sep 17 17:45:45.595115 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:45.595040    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwbl5\" (UniqueName: \"kubernetes.io/projected/84fe36a3-293a-4285-aaf5-00ee54c8f157-kube-api-access-nwbl5\") pod \"84fe36a3-293a-4285-aaf5-00ee54c8f157\" (UID: \"84fe36a3-293a-4285-aaf5-00ee54c8f157\") "
Sep 17 17:45:45.595115 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:45.595100    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84fe36a3-293a-4285-aaf5-00ee54c8f157-tls-certs\") pod \"84fe36a3-293a-4285-aaf5-00ee54c8f157\" (UID: \"84fe36a3-293a-4285-aaf5-00ee54c8f157\") "
Sep 17 17:45:45.597384 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:45.597349    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84fe36a3-293a-4285-aaf5-00ee54c8f157-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84fe36a3-293a-4285-aaf5-00ee54c8f157" (UID: "84fe36a3-293a-4285-aaf5-00ee54c8f157"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:45:45.597507 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:45.597388    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fe36a3-293a-4285-aaf5-00ee54c8f157-kube-api-access-nwbl5" (OuterVolumeSpecName: "kube-api-access-nwbl5") pod "84fe36a3-293a-4285-aaf5-00ee54c8f157" (UID: "84fe36a3-293a-4285-aaf5-00ee54c8f157"). InnerVolumeSpecName "kube-api-access-nwbl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:45:45.696315 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:45.696270    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84fe36a3-293a-4285-aaf5-00ee54c8f157-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:45:45.696315 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:45.696322    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwbl5\" (UniqueName: \"kubernetes.io/projected/84fe36a3-293a-4285-aaf5-00ee54c8f157-kube-api-access-nwbl5\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:45:46.215451 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:46.215410    2581 generic.go:356] "Generic (PLEG): container finished" podID="84fe36a3-293a-4285-aaf5-00ee54c8f157" containerID="0b0007704f172717beb39dba4f892f3ce9b8488923d0729a1033a2a24a75dcf9" exitCode=0
Sep 17 17:45:46.215647 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:46.215456    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"
Sep 17 17:45:46.215647 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:46.215495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn" event={"ID":"84fe36a3-293a-4285-aaf5-00ee54c8f157","Type":"ContainerDied","Data":"0b0007704f172717beb39dba4f892f3ce9b8488923d0729a1033a2a24a75dcf9"}
Sep 17 17:45:46.215647 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:46.215552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn" event={"ID":"84fe36a3-293a-4285-aaf5-00ee54c8f157","Type":"ContainerDied","Data":"8ef4d72ff184506e4ccf557fab30cecc5c5e9139d01e531b10c0fe77e6024142"}
Sep 17 17:45:46.215647 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:46.215573    2581 scope.go:117] "RemoveContainer" containerID="0b0007704f172717beb39dba4f892f3ce9b8488923d0729a1033a2a24a75dcf9"
Sep 17 17:45:46.224916 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:46.224900    2581 scope.go:117] "RemoveContainer" containerID="0b0007704f172717beb39dba4f892f3ce9b8488923d0729a1033a2a24a75dcf9"
Sep 17 17:45:46.225190 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:45:46.225170    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0007704f172717beb39dba4f892f3ce9b8488923d0729a1033a2a24a75dcf9\": container with ID starting with 0b0007704f172717beb39dba4f892f3ce9b8488923d0729a1033a2a24a75dcf9 not found: ID does not exist" containerID="0b0007704f172717beb39dba4f892f3ce9b8488923d0729a1033a2a24a75dcf9"
Sep 17 17:45:46.225263 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:46.225204    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0007704f172717beb39dba4f892f3ce9b8488923d0729a1033a2a24a75dcf9"} err="failed to get container status \"0b0007704f172717beb39dba4f892f3ce9b8488923d0729a1033a2a24a75dcf9\": rpc error: code = NotFound desc = could not find container \"0b0007704f172717beb39dba4f892f3ce9b8488923d0729a1033a2a24a75dcf9\": container with ID starting with 0b0007704f172717beb39dba4f892f3ce9b8488923d0729a1033a2a24a75dcf9 not found: ID does not exist"
Sep 17 17:45:46.255396 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:46.255372    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"]
Sep 17 17:45:46.259520 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:46.259501    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schedzztn"]
Sep 17 17:45:46.836545 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:46.836511    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84fe36a3-293a-4285-aaf5-00ee54c8f157" path="/var/lib/kubelet/pods/84fe36a3-293a-4285-aaf5-00ee54c8f157/volumes"
Sep 17 17:45:47.039775 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:47.039748    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:45:47.047852 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:45:47.047820    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:46:09.916012 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:09.915980    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"]
Sep 17 17:46:09.916480 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:09.916331    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84fe36a3-293a-4285-aaf5-00ee54c8f157" containerName="main"
Sep 17 17:46:09.916480 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:09.916342    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="84fe36a3-293a-4285-aaf5-00ee54c8f157" containerName="main"
Sep 17 17:46:09.916480 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:09.916399    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="84fe36a3-293a-4285-aaf5-00ee54c8f157" containerName="main"
Sep 17 17:46:09.918230 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:09.918213    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"
Sep 17 17:46:09.921917 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:09.921897    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-csz85\""
Sep 17 17:46:09.924679 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:09.924659    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-nwhht\""
Sep 17 17:46:09.924748 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:09.924672    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 17:46:09.924748 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:09.924706    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 17:46:09.927171 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:09.927149    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 17:46:09.933529 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:09.933506    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"]
Sep 17 17:46:10.079199 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:10.079173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqxhw\" (UniqueName: \"kubernetes.io/projected/e4e90a72-ca4e-4496-a68e-acd358f8b5d5-kube-api-access-kqxhw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6\" (UID: \"e4e90a72-ca4e-4496-a68e-acd358f8b5d5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"
Sep 17 17:46:10.079305 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:10.079219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e90a72-ca4e-4496-a68e-acd358f8b5d5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6\" (UID: \"e4e90a72-ca4e-4496-a68e-acd358f8b5d5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"
Sep 17 17:46:10.180266 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:10.180210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqxhw\" (UniqueName: \"kubernetes.io/projected/e4e90a72-ca4e-4496-a68e-acd358f8b5d5-kube-api-access-kqxhw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6\" (UID: \"e4e90a72-ca4e-4496-a68e-acd358f8b5d5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"
Sep 17 17:46:10.180266 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:10.180252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e90a72-ca4e-4496-a68e-acd358f8b5d5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6\" (UID: \"e4e90a72-ca4e-4496-a68e-acd358f8b5d5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"
Sep 17 17:46:10.182724 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:10.182703    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e90a72-ca4e-4496-a68e-acd358f8b5d5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6\" (UID: \"e4e90a72-ca4e-4496-a68e-acd358f8b5d5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"
Sep 17 17:46:10.190830 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:10.190806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqxhw\" (UniqueName: \"kubernetes.io/projected/e4e90a72-ca4e-4496-a68e-acd358f8b5d5-kube-api-access-kqxhw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6\" (UID: \"e4e90a72-ca4e-4496-a68e-acd358f8b5d5\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"
Sep 17 17:46:10.228382 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:10.228360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"
Sep 17 17:46:10.517440 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:10.517403    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"]
Sep 17 17:46:10.518818 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:46:10.518787    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e90a72_ca4e_4496_a68e_acd358f8b5d5.slice/crio-3744feaf26495a47903e16331d0bfe4be285d3af24fbb725504effee9374eeeb WatchSource:0}: Error finding container 3744feaf26495a47903e16331d0bfe4be285d3af24fbb725504effee9374eeeb: Status 404 returned error can't find the container with id 3744feaf26495a47903e16331d0bfe4be285d3af24fbb725504effee9374eeeb
Sep 17 17:46:11.335831 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:11.335783    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6" event={"ID":"e4e90a72-ca4e-4496-a68e-acd358f8b5d5","Type":"ContainerStarted","Data":"0c3ca90f9b997c43f710c93394c825c5104fa38cd08a47e0284c230eb4138182"}
Sep 17 17:46:11.336287 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:11.335839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6" event={"ID":"e4e90a72-ca4e-4496-a68e-acd358f8b5d5","Type":"ContainerStarted","Data":"3744feaf26495a47903e16331d0bfe4be285d3af24fbb725504effee9374eeeb"}
Sep 17 17:46:11.336287 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:11.335869    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"
Sep 17 17:46:11.359278 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:11.359223    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6" podStartSLOduration=2.359205444 podStartE2EDuration="2.359205444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:11.358904081 +0000 UTC m=+3850.781841944" watchObservedRunningTime="2026-09-17 17:46:11.359205444 +0000 UTC m=+3850.782143308"
Sep 17 17:46:31.498329 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:31.498248    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"]
Sep 17 17:46:42.342003 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:42.341940    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"
Sep 17 17:46:44.803658 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:44.803620    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"]
Sep 17 17:46:46.520757 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:46.520691    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main" containerID="cri-o://e7fdff9e0d1fa49ec5111add4a29c109ad0c4250f29d8edd0d3d953f4bc3541e" gracePeriod=45
Sep 17 17:46:47.041084 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.041046    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 17 17:46:47.183249 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.183225    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:46:47.249181 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.249151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c956d0ea-2a22-47c6-af38-2bcc521d2746-tls-certs\") pod \"c956d0ea-2a22-47c6-af38-2bcc521d2746\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") "
Sep 17 17:46:47.249317 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.249189    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-tmp-dir\") pod \"c956d0ea-2a22-47c6-af38-2bcc521d2746\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") "
Sep 17 17:46:47.249317 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.249219    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-kserve-provision-location\") pod \"c956d0ea-2a22-47c6-af38-2bcc521d2746\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") "
Sep 17 17:46:47.249447 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.249346    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7lxn\" (UniqueName: \"kubernetes.io/projected/c956d0ea-2a22-47c6-af38-2bcc521d2746-kube-api-access-n7lxn\") pod \"c956d0ea-2a22-47c6-af38-2bcc521d2746\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") "
Sep 17 17:46:47.249447 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.249429    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-model-cache\") pod \"c956d0ea-2a22-47c6-af38-2bcc521d2746\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") "
Sep 17 17:46:47.249561 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.249495    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-dshm\") pod \"c956d0ea-2a22-47c6-af38-2bcc521d2746\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") "
Sep 17 17:46:47.249561 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.249518    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-home\") pod \"c956d0ea-2a22-47c6-af38-2bcc521d2746\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") "
Sep 17 17:46:47.249561 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.249546    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c956d0ea-2a22-47c6-af38-2bcc521d2746-cabundle-cert\") pod \"c956d0ea-2a22-47c6-af38-2bcc521d2746\" (UID: \"c956d0ea-2a22-47c6-af38-2bcc521d2746\") "
Sep 17 17:46:47.249747 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.249711    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-model-cache" (OuterVolumeSpecName: "model-cache") pod "c956d0ea-2a22-47c6-af38-2bcc521d2746" (UID: "c956d0ea-2a22-47c6-af38-2bcc521d2746"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:47.249895 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.249874    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:46:47.250070 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.250047    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c956d0ea-2a22-47c6-af38-2bcc521d2746-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c956d0ea-2a22-47c6-af38-2bcc521d2746" (UID: "c956d0ea-2a22-47c6-af38-2bcc521d2746"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:46:47.250282 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.250242    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-home" (OuterVolumeSpecName: "home") pod "c956d0ea-2a22-47c6-af38-2bcc521d2746" (UID: "c956d0ea-2a22-47c6-af38-2bcc521d2746"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:47.252001 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.251953    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c956d0ea-2a22-47c6-af38-2bcc521d2746-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c956d0ea-2a22-47c6-af38-2bcc521d2746" (UID: "c956d0ea-2a22-47c6-af38-2bcc521d2746"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:46:47.252158 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.251995    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c956d0ea-2a22-47c6-af38-2bcc521d2746-kube-api-access-n7lxn" (OuterVolumeSpecName: "kube-api-access-n7lxn") pod "c956d0ea-2a22-47c6-af38-2bcc521d2746" (UID: "c956d0ea-2a22-47c6-af38-2bcc521d2746"). InnerVolumeSpecName "kube-api-access-n7lxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:46:47.252389 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.252360    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-dshm" (OuterVolumeSpecName: "dshm") pod "c956d0ea-2a22-47c6-af38-2bcc521d2746" (UID: "c956d0ea-2a22-47c6-af38-2bcc521d2746"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:47.260913 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.260865    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c956d0ea-2a22-47c6-af38-2bcc521d2746" (UID: "c956d0ea-2a22-47c6-af38-2bcc521d2746"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:47.273477 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.273452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c956d0ea-2a22-47c6-af38-2bcc521d2746" (UID: "c956d0ea-2a22-47c6-af38-2bcc521d2746"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:47.351226 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.351202    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:46:47.351226 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.351221    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:46:47.351360 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.351230    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c956d0ea-2a22-47c6-af38-2bcc521d2746-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:46:47.351360 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.351241    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c956d0ea-2a22-47c6-af38-2bcc521d2746-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:46:47.351360 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.351248    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:46:47.351360 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.351256    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c956d0ea-2a22-47c6-af38-2bcc521d2746-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:46:47.351360 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.351264    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7lxn\" (UniqueName: \"kubernetes.io/projected/c956d0ea-2a22-47c6-af38-2bcc521d2746-kube-api-access-n7lxn\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:46:47.516662 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.516599    2581 generic.go:356] "Generic (PLEG): container finished" podID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerID="e7fdff9e0d1fa49ec5111add4a29c109ad0c4250f29d8edd0d3d953f4bc3541e" exitCode=0
Sep 17 17:46:47.516662 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.516656    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" event={"ID":"c956d0ea-2a22-47c6-af38-2bcc521d2746","Type":"ContainerDied","Data":"e7fdff9e0d1fa49ec5111add4a29c109ad0c4250f29d8edd0d3d953f4bc3541e"}
Sep 17 17:46:47.516790 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.516659    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"
Sep 17 17:46:47.516790 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.516699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl" event={"ID":"c956d0ea-2a22-47c6-af38-2bcc521d2746","Type":"ContainerDied","Data":"c1cfa753aa04f517bbe47ed6b760cad6aa1c9cedd8dfc7623b4ac717044c98d8"}
Sep 17 17:46:47.516790 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.516722    2581 scope.go:117] "RemoveContainer" containerID="e7fdff9e0d1fa49ec5111add4a29c109ad0c4250f29d8edd0d3d953f4bc3541e"
Sep 17 17:46:47.526462 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.526273    2581 scope.go:117] "RemoveContainer" containerID="0886ac8db7d3889a2c5758fe6f0e041303d59d0d498f8d0ecea4e6f52f849315"
Sep 17 17:46:47.533936 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.533920    2581 scope.go:117] "RemoveContainer" containerID="e7fdff9e0d1fa49ec5111add4a29c109ad0c4250f29d8edd0d3d953f4bc3541e"
Sep 17 17:46:47.534231 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:46:47.534214    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7fdff9e0d1fa49ec5111add4a29c109ad0c4250f29d8edd0d3d953f4bc3541e\": container with ID starting with e7fdff9e0d1fa49ec5111add4a29c109ad0c4250f29d8edd0d3d953f4bc3541e not found: ID does not exist" containerID="e7fdff9e0d1fa49ec5111add4a29c109ad0c4250f29d8edd0d3d953f4bc3541e"
Sep 17 17:46:47.534309 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.534241    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7fdff9e0d1fa49ec5111add4a29c109ad0c4250f29d8edd0d3d953f4bc3541e"} err="failed to get container status \"e7fdff9e0d1fa49ec5111add4a29c109ad0c4250f29d8edd0d3d953f4bc3541e\": rpc error: code = NotFound desc = could not find container \"e7fdff9e0d1fa49ec5111add4a29c109ad0c4250f29d8edd0d3d953f4bc3541e\": container with ID starting with e7fdff9e0d1fa49ec5111add4a29c109ad0c4250f29d8edd0d3d953f4bc3541e not found: ID does not exist"
Sep 17 17:46:47.534309 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.534264    2581 scope.go:117] "RemoveContainer" containerID="0886ac8db7d3889a2c5758fe6f0e041303d59d0d498f8d0ecea4e6f52f849315"
Sep 17 17:46:47.534542 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:46:47.534522    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0886ac8db7d3889a2c5758fe6f0e041303d59d0d498f8d0ecea4e6f52f849315\": container with ID starting with 0886ac8db7d3889a2c5758fe6f0e041303d59d0d498f8d0ecea4e6f52f849315 not found: ID does not exist" containerID="0886ac8db7d3889a2c5758fe6f0e041303d59d0d498f8d0ecea4e6f52f849315"
Sep 17 17:46:47.534584 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.534550    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0886ac8db7d3889a2c5758fe6f0e041303d59d0d498f8d0ecea4e6f52f849315"} err="failed to get container status \"0886ac8db7d3889a2c5758fe6f0e041303d59d0d498f8d0ecea4e6f52f849315\": rpc error: code = NotFound desc = could not find container \"0886ac8db7d3889a2c5758fe6f0e041303d59d0d498f8d0ecea4e6f52f849315\": container with ID starting with 0886ac8db7d3889a2c5758fe6f0e041303d59d0d498f8d0ecea4e6f52f849315 not found: ID does not exist"
Sep 17 17:46:47.549217 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.549196    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"]
Sep 17 17:46:47.553413 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:47.553391    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-d8d857656-5c5gl"]
Sep 17 17:46:48.836394 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:48.836341    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" path="/var/lib/kubelet/pods/c956d0ea-2a22-47c6-af38-2bcc521d2746/volumes"
Sep 17 17:46:52.342469 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:52.342430    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6" podUID="e4e90a72-ca4e-4496-a68e-acd358f8b5d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:46:59.822995 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:46:59.822919    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6" podUID="e4e90a72-ca4e-4496-a68e-acd358f8b5d5" containerName="main" containerID="cri-o://0c3ca90f9b997c43f710c93394c825c5104fa38cd08a47e0284c230eb4138182" gracePeriod=45
Sep 17 17:47:02.341947 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:02.341896    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6" podUID="e4e90a72-ca4e-4496-a68e-acd358f8b5d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:47:03.072318 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:03.072288    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:47:03.079148 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:03.079126    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tr5db_c4158398-2987-4869-9185-3d8962e8ee54/ovn-acl-logging/0.log"
Sep 17 17:47:04.454825 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:04.454784    2581 scope.go:117] "RemoveContainer" containerID="b179b06b1edf18f2c80b9a3a806cd10319795ba8ff4c6e3c38e3f7db19f12799"
Sep 17 17:47:12.342371 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:12.342328    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6" podUID="e4e90a72-ca4e-4496-a68e-acd358f8b5d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:47:12.342876 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:12.342443    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"
Sep 17 17:47:22.342098 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:22.342051    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6" podUID="e4e90a72-ca4e-4496-a68e-acd358f8b5d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:47:30.053996 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.053954    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"
Sep 17 17:47:30.182443 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.182386    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e90a72-ca4e-4496-a68e-acd358f8b5d5-tls-certs\") pod \"e4e90a72-ca4e-4496-a68e-acd358f8b5d5\" (UID: \"e4e90a72-ca4e-4496-a68e-acd358f8b5d5\") "
Sep 17 17:47:30.182548 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.182447    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqxhw\" (UniqueName: \"kubernetes.io/projected/e4e90a72-ca4e-4496-a68e-acd358f8b5d5-kube-api-access-kqxhw\") pod \"e4e90a72-ca4e-4496-a68e-acd358f8b5d5\" (UID: \"e4e90a72-ca4e-4496-a68e-acd358f8b5d5\") "
Sep 17 17:47:30.184617 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.184589    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e90a72-ca4e-4496-a68e-acd358f8b5d5-kube-api-access-kqxhw" (OuterVolumeSpecName: "kube-api-access-kqxhw") pod "e4e90a72-ca4e-4496-a68e-acd358f8b5d5" (UID: "e4e90a72-ca4e-4496-a68e-acd358f8b5d5"). InnerVolumeSpecName "kube-api-access-kqxhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:47:30.184720 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.184658    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e90a72-ca4e-4496-a68e-acd358f8b5d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4e90a72-ca4e-4496-a68e-acd358f8b5d5" (UID: "e4e90a72-ca4e-4496-a68e-acd358f8b5d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:47:30.283288 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.283267    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4e90a72-ca4e-4496-a68e-acd358f8b5d5-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:47:30.283403 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.283290    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqxhw\" (UniqueName: \"kubernetes.io/projected/e4e90a72-ca4e-4496-a68e-acd358f8b5d5-kube-api-access-kqxhw\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:47:30.724695 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.724658    2581 generic.go:356] "Generic (PLEG): container finished" podID="e4e90a72-ca4e-4496-a68e-acd358f8b5d5" containerID="0c3ca90f9b997c43f710c93394c825c5104fa38cd08a47e0284c230eb4138182" exitCode=0
Sep 17 17:47:30.724861 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.724707    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"
Sep 17 17:47:30.724861 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.724715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6" event={"ID":"e4e90a72-ca4e-4496-a68e-acd358f8b5d5","Type":"ContainerDied","Data":"0c3ca90f9b997c43f710c93394c825c5104fa38cd08a47e0284c230eb4138182"}
Sep 17 17:47:30.724861 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.724768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6" event={"ID":"e4e90a72-ca4e-4496-a68e-acd358f8b5d5","Type":"ContainerDied","Data":"3744feaf26495a47903e16331d0bfe4be285d3af24fbb725504effee9374eeeb"}
Sep 17 17:47:30.724861 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.724799    2581 scope.go:117] "RemoveContainer" containerID="0c3ca90f9b997c43f710c93394c825c5104fa38cd08a47e0284c230eb4138182"
Sep 17 17:47:30.734007 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.733983    2581 scope.go:117] "RemoveContainer" containerID="0c3ca90f9b997c43f710c93394c825c5104fa38cd08a47e0284c230eb4138182"
Sep 17 17:47:30.734304 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:47:30.734281    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3ca90f9b997c43f710c93394c825c5104fa38cd08a47e0284c230eb4138182\": container with ID starting with 0c3ca90f9b997c43f710c93394c825c5104fa38cd08a47e0284c230eb4138182 not found: ID does not exist" containerID="0c3ca90f9b997c43f710c93394c825c5104fa38cd08a47e0284c230eb4138182"
Sep 17 17:47:30.734389 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.734317    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3ca90f9b997c43f710c93394c825c5104fa38cd08a47e0284c230eb4138182"} err="failed to get container status \"0c3ca90f9b997c43f710c93394c825c5104fa38cd08a47e0284c230eb4138182\": rpc error: code = NotFound desc = could not find container \"0c3ca90f9b997c43f710c93394c825c5104fa38cd08a47e0284c230eb4138182\": container with ID starting with 0c3ca90f9b997c43f710c93394c825c5104fa38cd08a47e0284c230eb4138182 not found: ID does not exist"
Sep 17 17:47:30.747633 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.747611    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"]
Sep 17 17:47:30.756657 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.756635    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9cpwc6"]
Sep 17 17:47:30.834948 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:47:30.834919    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4e90a72-ca4e-4496-a68e-acd358f8b5d5" path="/var/lib/kubelet/pods/e4e90a72-ca4e-4496-a68e-acd358f8b5d5/volumes"
Sep 17 17:48:07.996035 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:07.995993    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"]
Sep 17 17:48:07.996512 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:07.996343    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main"
Sep 17 17:48:07.996512 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:07.996355    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main"
Sep 17 17:48:07.996512 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:07.996373    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="storage-initializer"
Sep 17 17:48:07.996512 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:07.996379    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="storage-initializer"
Sep 17 17:48:07.996512 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:07.996394    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4e90a72-ca4e-4496-a68e-acd358f8b5d5" containerName="main"
Sep 17 17:48:07.996512 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:07.996400    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4e90a72-ca4e-4496-a68e-acd358f8b5d5" containerName="main"
Sep 17 17:48:07.996512 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:07.996459    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c956d0ea-2a22-47c6-af38-2bcc521d2746" containerName="main"
Sep 17 17:48:07.996512 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:07.996466    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4e90a72-ca4e-4496-a68e-acd358f8b5d5" containerName="main"
Sep 17 17:48:07.999193 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:07.999171    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.002040 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.002013    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 17:48:08.002914 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.002887    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 17:48:08.003065 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.002990    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 17:48:08.003065 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.003030    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-pjwpj\""
Sep 17 17:48:08.012592 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.012562    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"]
Sep 17 17:48:08.065602 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.065568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.065778 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.065605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.065778 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.065730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.065778 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.065768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.065956 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.065794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ljft\" (UniqueName: \"kubernetes.io/projected/4bfced90-86df-440a-8d14-836c6857a7bb-kube-api-access-6ljft\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.065956 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.065822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bfced90-86df-440a-8d14-836c6857a7bb-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.166591 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.166564    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.166704 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.166595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.166704 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.166652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.166704 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.166669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.166704 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.166689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ljft\" (UniqueName: \"kubernetes.io/projected/4bfced90-86df-440a-8d14-836c6857a7bb-kube-api-access-6ljft\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.166917 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.166712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bfced90-86df-440a-8d14-836c6857a7bb-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.167118 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.167096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.167201 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.167115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.167249 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.167218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.169155 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.169133    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.169300 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.169283    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bfced90-86df-440a-8d14-836c6857a7bb-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.184907 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.184885    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ljft\" (UniqueName: \"kubernetes.io/projected/4bfced90-86df-440a-8d14-836c6857a7bb-kube-api-access-6ljft\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58q89\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.310705 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.310647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:08.321559 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.321536    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"]
Sep 17 17:48:08.324494 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.324472    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"
Sep 17 17:48:08.327200 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.327169    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-djsr7\""
Sep 17 17:48:08.348022 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.347992    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"]
Sep 17 17:48:08.369371 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.369338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439e4664-629c-446c-871f-749c151b64da-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d\" (UID: \"439e4664-629c-446c-871f-749c151b64da\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"
Sep 17 17:48:08.369514 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.369491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n288n\" (UniqueName: \"kubernetes.io/projected/439e4664-629c-446c-871f-749c151b64da-kube-api-access-n288n\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d\" (UID: \"439e4664-629c-446c-871f-749c151b64da\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"
Sep 17 17:48:08.470318 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.470283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n288n\" (UniqueName: \"kubernetes.io/projected/439e4664-629c-446c-871f-749c151b64da-kube-api-access-n288n\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d\" (UID: \"439e4664-629c-446c-871f-749c151b64da\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"
Sep 17 17:48:08.470460 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.470340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439e4664-629c-446c-871f-749c151b64da-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d\" (UID: \"439e4664-629c-446c-871f-749c151b64da\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"
Sep 17 17:48:08.472947 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.472926    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439e4664-629c-446c-871f-749c151b64da-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d\" (UID: \"439e4664-629c-446c-871f-749c151b64da\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"
Sep 17 17:48:08.491068 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.491044    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n288n\" (UniqueName: \"kubernetes.io/projected/439e4664-629c-446c-871f-749c151b64da-kube-api-access-n288n\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d\" (UID: \"439e4664-629c-446c-871f-749c151b64da\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"
Sep 17 17:48:08.612070 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.611887    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"]
Sep 17 17:48:08.614794 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:48:08.614766    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfced90_86df_440a_8d14_836c6857a7bb.slice/crio-1a8d77d43d465ad2aad99c6046fb954c9f7839d6024db25974f80d181b5d296d WatchSource:0}: Error finding container 1a8d77d43d465ad2aad99c6046fb954c9f7839d6024db25974f80d181b5d296d: Status 404 returned error can't find the container with id 1a8d77d43d465ad2aad99c6046fb954c9f7839d6024db25974f80d181b5d296d
Sep 17 17:48:08.644584 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.644559    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"
Sep 17 17:48:08.913729 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.913646    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89" event={"ID":"4bfced90-86df-440a-8d14-836c6857a7bb","Type":"ContainerStarted","Data":"68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f"}
Sep 17 17:48:08.913729 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.913703    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89" event={"ID":"4bfced90-86df-440a-8d14-836c6857a7bb","Type":"ContainerStarted","Data":"1a8d77d43d465ad2aad99c6046fb954c9f7839d6024db25974f80d181b5d296d"}
Sep 17 17:48:08.920475 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.920451    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"]
Sep 17 17:48:08.923736 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:48:08.923711    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439e4664_629c_446c_871f_749c151b64da.slice/crio-c857d01d948c31f0deaa53c03eac33e516ffe22471c0e941d7ea53c3ee7c471e WatchSource:0}: Error finding container c857d01d948c31f0deaa53c03eac33e516ffe22471c0e941d7ea53c3ee7c471e: Status 404 returned error can't find the container with id c857d01d948c31f0deaa53c03eac33e516ffe22471c0e941d7ea53c3ee7c471e
Sep 17 17:48:08.941629 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:08.941583    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89" podStartSLOduration=1.9415679030000002 podStartE2EDuration="1.941567903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:08.938239505 +0000 UTC m=+3968.361177371" watchObservedRunningTime="2026-09-17 17:48:08.941567903 +0000 UTC m=+3968.364505814"
Sep 17 17:48:09.311693 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:09.311659    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:09.921319 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:09.921287    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d" event={"ID":"439e4664-629c-446c-871f-749c151b64da","Type":"ContainerStarted","Data":"e50adf5c8c621fce94954b2e65e0718f9f5ddfa20e229ee8e17bcfbf73eb688f"}
Sep 17 17:48:09.921594 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:09.921347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d" event={"ID":"439e4664-629c-446c-871f-749c151b64da","Type":"ContainerStarted","Data":"c857d01d948c31f0deaa53c03eac33e516ffe22471c0e941d7ea53c3ee7c471e"}
Sep 17 17:48:09.921594 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:09.921538    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"
Sep 17 17:48:09.947321 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:09.947276    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d" podStartSLOduration=1.9472634389999999 podStartE2EDuration="1.947263439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:09.94537654 +0000 UTC m=+3969.368314402" watchObservedRunningTime="2026-09-17 17:48:09.947263439 +0000 UTC m=+3969.370201301"
Sep 17 17:48:18.311279 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:18.311235    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:18.323958 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:18.323927    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:18.975816 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:18.975781    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:41.932642 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:41.932614    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"
Sep 17 17:48:50.236256 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.236220    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"]
Sep 17 17:48:50.259415 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.259389    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"]
Sep 17 17:48:50.267377 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:48:50.267349    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:48:50.267377 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:48:50.267377 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:48:50.267377 ip-10-0-140-49 kubenswrapper[2581]:  > containerID="68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f" cmd=["/bin/sleep","15"]
Sep 17 17:48:50.267592 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.267393    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89" message="" err=<
Sep 17 17:48:50.267592 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:48:50.267592 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:48:50.267592 ip-10-0-140-49 kubenswrapper[2581]:  >
Sep 17 17:48:50.267592 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:48:50.267406    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:48:50.267592 ip-10-0-140-49 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:48:50.267592 ip-10-0-140-49 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 17:48:50.267592 ip-10-0-140-49 kubenswrapper[2581]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89" podUID="4bfced90-86df-440a-8d14-836c6857a7bb" containerName="main" containerID="cri-o://68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f"
Sep 17 17:48:50.267592 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.267444    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89" podUID="4bfced90-86df-440a-8d14-836c6857a7bb" containerName="main" containerID="cri-o://68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f" gracePeriod=60
Sep 17 17:48:50.607396 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.607366    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:50.690573 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.690543    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-model-cache\") pod \"4bfced90-86df-440a-8d14-836c6857a7bb\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") "
Sep 17 17:48:50.690717 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.690598    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ljft\" (UniqueName: \"kubernetes.io/projected/4bfced90-86df-440a-8d14-836c6857a7bb-kube-api-access-6ljft\") pod \"4bfced90-86df-440a-8d14-836c6857a7bb\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") "
Sep 17 17:48:50.690717 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.690633    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bfced90-86df-440a-8d14-836c6857a7bb-tls-certs\") pod \"4bfced90-86df-440a-8d14-836c6857a7bb\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") "
Sep 17 17:48:50.690717 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.690683    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-dshm\") pod \"4bfced90-86df-440a-8d14-836c6857a7bb\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") "
Sep 17 17:48:50.690717 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.690704    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-home\") pod \"4bfced90-86df-440a-8d14-836c6857a7bb\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") "
Sep 17 17:48:50.690919 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.690735    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-tmp-dir\") pod \"4bfced90-86df-440a-8d14-836c6857a7bb\" (UID: \"4bfced90-86df-440a-8d14-836c6857a7bb\") "
Sep 17 17:48:50.690919 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.690816    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "4bfced90-86df-440a-8d14-836c6857a7bb" (UID: "4bfced90-86df-440a-8d14-836c6857a7bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:50.691038 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.690992    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-home" (OuterVolumeSpecName: "home") pod "4bfced90-86df-440a-8d14-836c6857a7bb" (UID: "4bfced90-86df-440a-8d14-836c6857a7bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:50.691109 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.691080    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:48:50.691109 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.691098    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:48:50.691109 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.691088    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4bfced90-86df-440a-8d14-836c6857a7bb" (UID: "4bfced90-86df-440a-8d14-836c6857a7bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:50.693007 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.692983    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-dshm" (OuterVolumeSpecName: "dshm") pod "4bfced90-86df-440a-8d14-836c6857a7bb" (UID: "4bfced90-86df-440a-8d14-836c6857a7bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:48:50.693103 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.693027    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfced90-86df-440a-8d14-836c6857a7bb-kube-api-access-6ljft" (OuterVolumeSpecName: "kube-api-access-6ljft") pod "4bfced90-86df-440a-8d14-836c6857a7bb" (UID: "4bfced90-86df-440a-8d14-836c6857a7bb"). InnerVolumeSpecName "kube-api-access-6ljft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:48:50.693103 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.693031    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bfced90-86df-440a-8d14-836c6857a7bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4bfced90-86df-440a-8d14-836c6857a7bb" (UID: "4bfced90-86df-440a-8d14-836c6857a7bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:48:50.792125 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.792071    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ljft\" (UniqueName: \"kubernetes.io/projected/4bfced90-86df-440a-8d14-836c6857a7bb-kube-api-access-6ljft\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:48:50.792125 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.792094    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bfced90-86df-440a-8d14-836c6857a7bb-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:48:50.792125 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.792106    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:48:50.792279 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:50.792134    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bfced90-86df-440a-8d14-836c6857a7bb-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:48:51.123338 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:51.123274    2581 generic.go:356] "Generic (PLEG): container finished" podID="4bfced90-86df-440a-8d14-836c6857a7bb" containerID="68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f" exitCode=0
Sep 17 17:48:51.123338 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:51.123324    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"
Sep 17 17:48:51.123338 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:51.123327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89" event={"ID":"4bfced90-86df-440a-8d14-836c6857a7bb","Type":"ContainerDied","Data":"68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f"}
Sep 17 17:48:51.123507 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:51.123378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89" event={"ID":"4bfced90-86df-440a-8d14-836c6857a7bb","Type":"ContainerDied","Data":"1a8d77d43d465ad2aad99c6046fb954c9f7839d6024db25974f80d181b5d296d"}
Sep 17 17:48:51.123507 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:51.123407    2581 scope.go:117] "RemoveContainer" containerID="68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f"
Sep 17 17:48:51.131976 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:51.131946    2581 scope.go:117] "RemoveContainer" containerID="68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f"
Sep 17 17:48:51.132302 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:48:51.132283    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f\": container with ID starting with 68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f not found: ID does not exist" containerID="68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f"
Sep 17 17:48:51.132350 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:51.132314    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f"} err="failed to get container status \"68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f\": rpc error: code = NotFound desc = could not find container \"68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f\": container with ID starting with 68fe2c73c82baaf5b6f35203c5e6d99827cf7611e7db5220853146430f11ef5f not found: ID does not exist"
Sep 17 17:48:51.150562 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:51.150540    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"]
Sep 17 17:48:51.156836 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:51.156816    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58q89"]
Sep 17 17:48:51.933222 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:51.933182    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d" podUID="439e4664-629c-446c-871f-749c151b64da" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:48:52.836285 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:48:52.836242    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bfced90-86df-440a-8d14-836c6857a7bb" path="/var/lib/kubelet/pods/4bfced90-86df-440a-8d14-836c6857a7bb/volumes"
Sep 17 17:49:01.932903 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:01.932857    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d" podUID="439e4664-629c-446c-871f-749c151b64da" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:49:05.254942 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:05.254884    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d" podUID="439e4664-629c-446c-871f-749c151b64da" containerName="main" containerID="cri-o://e50adf5c8c621fce94954b2e65e0718f9f5ddfa20e229ee8e17bcfbf73eb688f" gracePeriod=45
Sep 17 17:49:11.932438 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:11.932398    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d" podUID="439e4664-629c-446c-871f-749c151b64da" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:49:11.932820 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:11.932502    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"
Sep 17 17:49:21.932957 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:21.932845    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d" podUID="439e4664-629c-446c-871f-749c151b64da" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:49:28.932278 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.932238    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"]
Sep 17 17:49:28.932663 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.932624    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bfced90-86df-440a-8d14-836c6857a7bb" containerName="main"
Sep 17 17:49:28.932663 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.932640    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bfced90-86df-440a-8d14-836c6857a7bb" containerName="main"
Sep 17 17:49:28.932746 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.932709    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bfced90-86df-440a-8d14-836c6857a7bb" containerName="main"
Sep 17 17:49:28.935725 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.935709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:28.943009 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.942985    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:49:28.943160 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.942994    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 17:49:28.943160 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.943072    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 17:49:28.943796 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.943778    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 17:49:28.943898 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.943802    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-rsrk2\""
Sep 17 17:49:28.943898 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.943820    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 17:49:28.949959 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.949937    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"]
Sep 17 17:49:28.967114 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.967084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5a2096-eceb-4a95-82a6-36f343eebd4f-tls-certs\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:28.967227 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.967115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmbnp\" (UniqueName: \"kubernetes.io/projected/3d5a2096-eceb-4a95-82a6-36f343eebd4f-kube-api-access-bmbnp\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:28.967227 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.967151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-home\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:28.967227 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.967167    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d5a2096-eceb-4a95-82a6-36f343eebd4f-cabundle-cert\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:28.967227 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.967197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-model-cache\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:28.967388 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.967269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-tmp-dir\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:28.967388 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.967304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-kserve-provision-location\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:28.967388 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:28.967339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-dshm\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.067774 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.067739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-home\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.067774 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.067772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d5a2096-eceb-4a95-82a6-36f343eebd4f-cabundle-cert\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.068024 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.067795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-model-cache\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.068024 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.067844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-tmp-dir\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.068024 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.067878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-kserve-provision-location\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.068024 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.067918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-dshm\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.068024 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.067949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5a2096-eceb-4a95-82a6-36f343eebd4f-tls-certs\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.068024 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.067992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmbnp\" (UniqueName: \"kubernetes.io/projected/3d5a2096-eceb-4a95-82a6-36f343eebd4f-kube-api-access-bmbnp\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.068354 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.068220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-home\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.068354 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.068247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-model-cache\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.068465 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.068357    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-kserve-provision-location\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.068465 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.068430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-tmp-dir\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.068680 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.068660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d5a2096-eceb-4a95-82a6-36f343eebd4f-cabundle-cert\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.070314 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.070289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-dshm\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.070610 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.070593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5a2096-eceb-4a95-82a6-36f343eebd4f-tls-certs\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.080853 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.080833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmbnp\" (UniqueName: \"kubernetes.io/projected/3d5a2096-eceb-4a95-82a6-36f343eebd4f-kube-api-access-bmbnp\") pod \"prestop-hook-test-kserve-86565fd9b-8d9g2\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.245842 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.245812    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:29.528924 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:29.528894    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"]
Sep 17 17:49:29.529564 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:49:29.529539    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5a2096_eceb_4a95_82a6_36f343eebd4f.slice/crio-19c616c3602ebfe462446334ba29bfab09ba7bb5f34b53c8ab184d849eec2290 WatchSource:0}: Error finding container 19c616c3602ebfe462446334ba29bfab09ba7bb5f34b53c8ab184d849eec2290: Status 404 returned error can't find the container with id 19c616c3602ebfe462446334ba29bfab09ba7bb5f34b53c8ab184d849eec2290
Sep 17 17:49:30.313770 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:30.313730    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2" event={"ID":"3d5a2096-eceb-4a95-82a6-36f343eebd4f","Type":"ContainerStarted","Data":"312fa0f846f37e450422dd98f45668ce6f772fbe93b462a1ad3580ab1d118440"}
Sep 17 17:49:30.314292 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:30.313784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2" event={"ID":"3d5a2096-eceb-4a95-82a6-36f343eebd4f","Type":"ContainerStarted","Data":"19c616c3602ebfe462446334ba29bfab09ba7bb5f34b53c8ab184d849eec2290"}
Sep 17 17:49:31.933252 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:31.933215    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d" podUID="439e4664-629c-446c-871f-749c151b64da" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:49:32.324774 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:32.324739    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d5a2096-eceb-4a95-82a6-36f343eebd4f" containerID="312fa0f846f37e450422dd98f45668ce6f772fbe93b462a1ad3580ab1d118440" exitCode=0
Sep 17 17:49:32.324948 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:32.324813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2" event={"ID":"3d5a2096-eceb-4a95-82a6-36f343eebd4f","Type":"ContainerDied","Data":"312fa0f846f37e450422dd98f45668ce6f772fbe93b462a1ad3580ab1d118440"}
Sep 17 17:49:33.332432 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:33.332392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2" event={"ID":"3d5a2096-eceb-4a95-82a6-36f343eebd4f","Type":"ContainerStarted","Data":"ee3b843c6772464e0abb39bcdef0986c235e322144942b31d242b8785337e1e1"}
Sep 17 17:49:34.245925 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:34.245893    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:35.510166 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:35.510136    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"
Sep 17 17:49:35.514175 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:35.514117    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439e4664-629c-446c-871f-749c151b64da-tls-certs\") pod \"439e4664-629c-446c-871f-749c151b64da\" (UID: \"439e4664-629c-446c-871f-749c151b64da\") "
Sep 17 17:49:35.516203 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:35.516176    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439e4664-629c-446c-871f-749c151b64da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "439e4664-629c-446c-871f-749c151b64da" (UID: "439e4664-629c-446c-871f-749c151b64da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:49:35.538101 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:35.538041    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2" podStartSLOduration=5.372695886 podStartE2EDuration="7.538019287s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.165323401s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:33.377861295 +0000 UTC m=+4052.800799184" watchObservedRunningTime="2026-09-17 17:49:35.538019287 +0000 UTC m=+4054.960957166"
Sep 17 17:49:35.615066 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:35.615029    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n288n\" (UniqueName: \"kubernetes.io/projected/439e4664-629c-446c-871f-749c151b64da-kube-api-access-n288n\") pod \"439e4664-629c-446c-871f-749c151b64da\" (UID: \"439e4664-629c-446c-871f-749c151b64da\") "
Sep 17 17:49:35.615265 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:35.615240    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439e4664-629c-446c-871f-749c151b64da-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:49:35.617219 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:35.617189    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439e4664-629c-446c-871f-749c151b64da-kube-api-access-n288n" (OuterVolumeSpecName: "kube-api-access-n288n") pod "439e4664-629c-446c-871f-749c151b64da" (UID: "439e4664-629c-446c-871f-749c151b64da"). InnerVolumeSpecName "kube-api-access-n288n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:49:35.715929 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:35.715889    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n288n\" (UniqueName: \"kubernetes.io/projected/439e4664-629c-446c-871f-749c151b64da-kube-api-access-n288n\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:49:36.348674 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:36.348638    2581 generic.go:356] "Generic (PLEG): container finished" podID="439e4664-629c-446c-871f-749c151b64da" containerID="e50adf5c8c621fce94954b2e65e0718f9f5ddfa20e229ee8e17bcfbf73eb688f" exitCode=0
Sep 17 17:49:36.348852 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:36.348690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d" event={"ID":"439e4664-629c-446c-871f-749c151b64da","Type":"ContainerDied","Data":"e50adf5c8c621fce94954b2e65e0718f9f5ddfa20e229ee8e17bcfbf73eb688f"}
Sep 17 17:49:36.348852 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:36.348697    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"
Sep 17 17:49:36.348852 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:36.348734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d" event={"ID":"439e4664-629c-446c-871f-749c151b64da","Type":"ContainerDied","Data":"c857d01d948c31f0deaa53c03eac33e516ffe22471c0e941d7ea53c3ee7c471e"}
Sep 17 17:49:36.348852 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:36.348757    2581 scope.go:117] "RemoveContainer" containerID="e50adf5c8c621fce94954b2e65e0718f9f5ddfa20e229ee8e17bcfbf73eb688f"
Sep 17 17:49:36.357159 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:36.357138    2581 scope.go:117] "RemoveContainer" containerID="e50adf5c8c621fce94954b2e65e0718f9f5ddfa20e229ee8e17bcfbf73eb688f"
Sep 17 17:49:36.357424 ip-10-0-140-49 kubenswrapper[2581]: E0917 17:49:36.357405    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50adf5c8c621fce94954b2e65e0718f9f5ddfa20e229ee8e17bcfbf73eb688f\": container with ID starting with e50adf5c8c621fce94954b2e65e0718f9f5ddfa20e229ee8e17bcfbf73eb688f not found: ID does not exist" containerID="e50adf5c8c621fce94954b2e65e0718f9f5ddfa20e229ee8e17bcfbf73eb688f"
Sep 17 17:49:36.357486 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:36.357438    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50adf5c8c621fce94954b2e65e0718f9f5ddfa20e229ee8e17bcfbf73eb688f"} err="failed to get container status \"e50adf5c8c621fce94954b2e65e0718f9f5ddfa20e229ee8e17bcfbf73eb688f\": rpc error: code = NotFound desc = could not find container \"e50adf5c8c621fce94954b2e65e0718f9f5ddfa20e229ee8e17bcfbf73eb688f\": container with ID starting with e50adf5c8c621fce94954b2e65e0718f9f5ddfa20e229ee8e17bcfbf73eb688f not found: ID does not exist"
Sep 17 17:49:36.378729 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:36.378701    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"]
Sep 17 17:49:36.383581 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:36.383559    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-t426d"]
Sep 17 17:49:36.837115 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:36.837081    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="439e4664-629c-446c-871f-749c151b64da" path="/var/lib/kubelet/pods/439e4664-629c-446c-871f-749c151b64da/volumes"
Sep 17 17:49:39.246135 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:39.246095    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:49:39.247824 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:39.247794    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2" podUID="3d5a2096-eceb-4a95-82a6-36f343eebd4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 17:49:47.469204 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:49:47.469164    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"]
Sep 17 17:50:02.492731 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:02.492691    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2" podUID="3d5a2096-eceb-4a95-82a6-36f343eebd4f" containerName="main" containerID="cri-o://ee3b843c6772464e0abb39bcdef0986c235e322144942b31d242b8785337e1e1" gracePeriod=45
Sep 17 17:50:47.675292 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:47.675268    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-86565fd9b-8d9g2_3d5a2096-eceb-4a95-82a6-36f343eebd4f/main/0.log"
Sep 17 17:50:47.675698 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:47.675672    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d5a2096-eceb-4a95-82a6-36f343eebd4f" containerID="ee3b843c6772464e0abb39bcdef0986c235e322144942b31d242b8785337e1e1" exitCode=137
Sep 17 17:50:47.675774 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:47.675751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2" event={"ID":"3d5a2096-eceb-4a95-82a6-36f343eebd4f","Type":"ContainerDied","Data":"ee3b843c6772464e0abb39bcdef0986c235e322144942b31d242b8785337e1e1"}
Sep 17 17:50:48.134350 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.134321    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-86565fd9b-8d9g2_3d5a2096-eceb-4a95-82a6-36f343eebd4f/main/0.log"
Sep 17 17:50:48.137031 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.137013    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:50:48.231070 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.231048    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmbnp\" (UniqueName: \"kubernetes.io/projected/3d5a2096-eceb-4a95-82a6-36f343eebd4f-kube-api-access-bmbnp\") pod \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") "
Sep 17 17:50:48.231192 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.231088    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5a2096-eceb-4a95-82a6-36f343eebd4f-tls-certs\") pod \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") "
Sep 17 17:50:48.231192 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.231111    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-kserve-provision-location\") pod \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") "
Sep 17 17:50:48.231192 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.231146    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-model-cache\") pod \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") "
Sep 17 17:50:48.231192 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.231169    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d5a2096-eceb-4a95-82a6-36f343eebd4f-cabundle-cert\") pod \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") "
Sep 17 17:50:48.231192 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.231191    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-dshm\") pod \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") "
Sep 17 17:50:48.231459 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.231220    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-tmp-dir\") pod \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") "
Sep 17 17:50:48.231520 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.231471    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d5a2096-eceb-4a95-82a6-36f343eebd4f" (UID: "3d5a2096-eceb-4a95-82a6-36f343eebd4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:48.231579 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.231541    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5a2096-eceb-4a95-82a6-36f343eebd4f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d5a2096-eceb-4a95-82a6-36f343eebd4f" (UID: "3d5a2096-eceb-4a95-82a6-36f343eebd4f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:48.231647 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.231604    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-home\") pod \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\" (UID: \"3d5a2096-eceb-4a95-82a6-36f343eebd4f\") "
Sep 17 17:50:48.231703 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.231669    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d5a2096-eceb-4a95-82a6-36f343eebd4f" (UID: "3d5a2096-eceb-4a95-82a6-36f343eebd4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:48.232011 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.231987    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d5a2096-eceb-4a95-82a6-36f343eebd4f-cabundle-cert\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:50:48.232112 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.232016    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-tmp-dir\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:50:48.232112 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.232030    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-model-cache\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:50:48.232112 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.232038    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-home" (OuterVolumeSpecName: "home") pod "3d5a2096-eceb-4a95-82a6-36f343eebd4f" (UID: "3d5a2096-eceb-4a95-82a6-36f343eebd4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:48.233484 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.233461    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-dshm" (OuterVolumeSpecName: "dshm") pod "3d5a2096-eceb-4a95-82a6-36f343eebd4f" (UID: "3d5a2096-eceb-4a95-82a6-36f343eebd4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:48.233739 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.233482    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d5a2096-eceb-4a95-82a6-36f343eebd4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d5a2096-eceb-4a95-82a6-36f343eebd4f" (UID: "3d5a2096-eceb-4a95-82a6-36f343eebd4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:48.233739 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.233594    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5a2096-eceb-4a95-82a6-36f343eebd4f-kube-api-access-bmbnp" (OuterVolumeSpecName: "kube-api-access-bmbnp") pod "3d5a2096-eceb-4a95-82a6-36f343eebd4f" (UID: "3d5a2096-eceb-4a95-82a6-36f343eebd4f"). InnerVolumeSpecName "kube-api-access-bmbnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:50:48.254477 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.254453    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d5a2096-eceb-4a95-82a6-36f343eebd4f" (UID: "3d5a2096-eceb-4a95-82a6-36f343eebd4f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:48.333194 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.333138    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmbnp\" (UniqueName: \"kubernetes.io/projected/3d5a2096-eceb-4a95-82a6-36f343eebd4f-kube-api-access-bmbnp\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:50:48.333194 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.333161    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5a2096-eceb-4a95-82a6-36f343eebd4f-tls-certs\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:50:48.333194 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.333172    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-kserve-provision-location\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:50:48.333194 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.333179    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-dshm\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:50:48.333194 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.333188    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d5a2096-eceb-4a95-82a6-36f343eebd4f-home\") on node \"ip-10-0-140-49.ec2.internal\" DevicePath \"\""
Sep 17 17:50:48.682590 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.682514    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-86565fd9b-8d9g2_3d5a2096-eceb-4a95-82a6-36f343eebd4f/main/0.log"
Sep 17 17:50:48.683026 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.682959    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"
Sep 17 17:50:48.683094 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.682953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2" event={"ID":"3d5a2096-eceb-4a95-82a6-36f343eebd4f","Type":"ContainerDied","Data":"19c616c3602ebfe462446334ba29bfab09ba7bb5f34b53c8ab184d849eec2290"}
Sep 17 17:50:48.683137 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.683124    2581 scope.go:117] "RemoveContainer" containerID="ee3b843c6772464e0abb39bcdef0986c235e322144942b31d242b8785337e1e1"
Sep 17 17:50:48.693204 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.693186    2581 scope.go:117] "RemoveContainer" containerID="312fa0f846f37e450422dd98f45668ce6f772fbe93b462a1ad3580ab1d118440"
Sep 17 17:50:48.709041 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.709008    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"]
Sep 17 17:50:48.714847 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.714821    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-86565fd9b-8d9g2"]
Sep 17 17:50:48.835928 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:50:48.835899    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d5a2096-eceb-4a95-82a6-36f343eebd4f" path="/var/lib/kubelet/pods/3d5a2096-eceb-4a95-82a6-36f343eebd4f/volumes"
Sep 17 17:51:22.454635 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:22.454580    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6l266_bd12b862-34c9-4adc-b3d3-178f564aea1e/manager/0.log"
Sep 17 17:51:22.617407 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:22.617379    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-prrwc_8b987118-e74d-4e45-b9f2-4587471513a2/manager/0.log"
Sep 17 17:51:25.117063 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.117024    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2krt9/must-gather-2pcjg"]
Sep 17 17:51:25.117473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.117357    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d5a2096-eceb-4a95-82a6-36f343eebd4f" containerName="storage-initializer"
Sep 17 17:51:25.117473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.117369    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d5a2096-eceb-4a95-82a6-36f343eebd4f" containerName="storage-initializer"
Sep 17 17:51:25.117473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.117384    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="439e4664-629c-446c-871f-749c151b64da" containerName="main"
Sep 17 17:51:25.117473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.117391    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="439e4664-629c-446c-871f-749c151b64da" containerName="main"
Sep 17 17:51:25.117473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.117400    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d5a2096-eceb-4a95-82a6-36f343eebd4f" containerName="main"
Sep 17 17:51:25.117473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.117406    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d5a2096-eceb-4a95-82a6-36f343eebd4f" containerName="main"
Sep 17 17:51:25.117473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.117471    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="439e4664-629c-446c-871f-749c151b64da" containerName="main"
Sep 17 17:51:25.117473 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.117478    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d5a2096-eceb-4a95-82a6-36f343eebd4f" containerName="main"
Sep 17 17:51:25.120461 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.120443    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2krt9/must-gather-2pcjg"
Sep 17 17:51:25.126521 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.126489    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2krt9\"/\"kube-root-ca.crt\""
Sep 17 17:51:25.126670 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.126502    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2krt9\"/\"openshift-service-ca.crt\""
Sep 17 17:51:25.127676 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.127483    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2krt9\"/\"default-dockercfg-9ssl8\""
Sep 17 17:51:25.147251 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.147226    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2krt9/must-gather-2pcjg"]
Sep 17 17:51:25.206921 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.206895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9s9b\" (UniqueName: \"kubernetes.io/projected/f0ef51c2-2ec6-4f10-938c-eb5e87712408-kube-api-access-c9s9b\") pod \"must-gather-2pcjg\" (UID: \"f0ef51c2-2ec6-4f10-938c-eb5e87712408\") " pod="openshift-must-gather-2krt9/must-gather-2pcjg"
Sep 17 17:51:25.207040 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.206946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f0ef51c2-2ec6-4f10-938c-eb5e87712408-must-gather-output\") pod \"must-gather-2pcjg\" (UID: \"f0ef51c2-2ec6-4f10-938c-eb5e87712408\") " pod="openshift-must-gather-2krt9/must-gather-2pcjg"
Sep 17 17:51:25.307928 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.307902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f0ef51c2-2ec6-4f10-938c-eb5e87712408-must-gather-output\") pod \"must-gather-2pcjg\" (UID: \"f0ef51c2-2ec6-4f10-938c-eb5e87712408\") " pod="openshift-must-gather-2krt9/must-gather-2pcjg"
Sep 17 17:51:25.308053 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.307956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9s9b\" (UniqueName: \"kubernetes.io/projected/f0ef51c2-2ec6-4f10-938c-eb5e87712408-kube-api-access-c9s9b\") pod \"must-gather-2pcjg\" (UID: \"f0ef51c2-2ec6-4f10-938c-eb5e87712408\") " pod="openshift-must-gather-2krt9/must-gather-2pcjg"
Sep 17 17:51:25.308282 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.308259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f0ef51c2-2ec6-4f10-938c-eb5e87712408-must-gather-output\") pod \"must-gather-2pcjg\" (UID: \"f0ef51c2-2ec6-4f10-938c-eb5e87712408\") " pod="openshift-must-gather-2krt9/must-gather-2pcjg"
Sep 17 17:51:25.329953 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.329923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9s9b\" (UniqueName: \"kubernetes.io/projected/f0ef51c2-2ec6-4f10-938c-eb5e87712408-kube-api-access-c9s9b\") pod \"must-gather-2pcjg\" (UID: \"f0ef51c2-2ec6-4f10-938c-eb5e87712408\") " pod="openshift-must-gather-2krt9/must-gather-2pcjg"
Sep 17 17:51:25.430152 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.430090    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2krt9/must-gather-2pcjg"
Sep 17 17:51:25.714595 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.714562    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2krt9/must-gather-2pcjg"]
Sep 17 17:51:25.717904 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:51:25.717871    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0ef51c2_2ec6_4f10_938c_eb5e87712408.slice/crio-37547f6ba9893029d39c18a2baba74e0f1fe0db45fe91ed074ff069e20eb1059 WatchSource:0}: Error finding container 37547f6ba9893029d39c18a2baba74e0f1fe0db45fe91ed074ff069e20eb1059: Status 404 returned error can't find the container with id 37547f6ba9893029d39c18a2baba74e0f1fe0db45fe91ed074ff069e20eb1059
Sep 17 17:51:25.719786 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.719766    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:51:25.863871 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:25.863844    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2krt9/must-gather-2pcjg" event={"ID":"f0ef51c2-2ec6-4f10-938c-eb5e87712408","Type":"ContainerStarted","Data":"37547f6ba9893029d39c18a2baba74e0f1fe0db45fe91ed074ff069e20eb1059"}
Sep 17 17:51:26.871952 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:26.871864    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2krt9/must-gather-2pcjg" event={"ID":"f0ef51c2-2ec6-4f10-938c-eb5e87712408","Type":"ContainerStarted","Data":"9695c48e0fa92907a62349b2f74bd7f5ee66e0adf9b537bf21dda37cd5787a83"}
Sep 17 17:51:26.871952 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:26.871912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2krt9/must-gather-2pcjg" event={"ID":"f0ef51c2-2ec6-4f10-938c-eb5e87712408","Type":"ContainerStarted","Data":"47bf18785894051a173dd4b9ea463510bdf7ef350b64909ccdb847ade2c5c8c6"}
Sep 17 17:51:28.573691 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:28.573656    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lgk7x_667acd31-9d7e-4010-bbb3-f18b17232f48/global-pull-secret-syncer/0.log"
Sep 17 17:51:28.775450 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:28.775416    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vscfj_21fa7670-4703-438a-9c6b-3caa76193e5b/konnectivity-agent/0.log"
Sep 17 17:51:28.884180 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:28.884100    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-49.ec2.internal_af9930b89e342a5bca676a2c71f00917/haproxy/0.log"
Sep 17 17:51:33.635590 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:33.635559    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-6l266_bd12b862-34c9-4adc-b3d3-178f564aea1e/manager/0.log"
Sep 17 17:51:33.848070 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:33.848036    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-prrwc_8b987118-e74d-4e45-b9f2-4587471513a2/manager/0.log"
Sep 17 17:51:35.635594 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:35.635561    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6/alertmanager/0.log"
Sep 17 17:51:35.687776 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:35.687747    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6/config-reloader/0.log"
Sep 17 17:51:35.731194 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:35.731152    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6/kube-rbac-proxy-web/0.log"
Sep 17 17:51:35.771950 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:35.771904    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6/kube-rbac-proxy/0.log"
Sep 17 17:51:35.810690 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:35.810657    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6/kube-rbac-proxy-metric/0.log"
Sep 17 17:51:35.854866 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:35.854825    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6/prom-label-proxy/0.log"
Sep 17 17:51:35.892609 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:35.892518    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f3525b5-bf6e-4ecd-bb27-185fd0f6fec6/init-config-reloader/0.log"
Sep 17 17:51:36.041226 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:36.041191    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-45n6z_72b72d14-1c90-49a1-adfe-8d9d62e9c7f4/kube-state-metrics/0.log"
Sep 17 17:51:36.120845 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:36.120819    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-45n6z_72b72d14-1c90-49a1-adfe-8d9d62e9c7f4/kube-rbac-proxy-main/0.log"
Sep 17 17:51:36.165230 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:36.165144    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-45n6z_72b72d14-1c90-49a1-adfe-8d9d62e9c7f4/kube-rbac-proxy-self/0.log"
Sep 17 17:51:36.252326 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:36.252295    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-pdfvq_9e0d9dc0-af8b-4ab5-8d8d-c020ee1209b5/monitoring-plugin/0.log"
Sep 17 17:51:36.499000 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:36.498945    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh4k5_2a27dbb7-6616-42f2-b562-b7aebfb49adb/node-exporter/0.log"
Sep 17 17:51:36.534026 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:36.533996    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh4k5_2a27dbb7-6616-42f2-b562-b7aebfb49adb/kube-rbac-proxy/0.log"
Sep 17 17:51:36.580701 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:36.580671    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vh4k5_2a27dbb7-6616-42f2-b562-b7aebfb49adb/init-textfile/0.log"
Sep 17 17:51:36.990288 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:36.990255    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-5lwqk_70256eb4-eec2-4d39-8420-5403098c7cbf/prometheus-operator/0.log"
Sep 17 17:51:37.033789 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.033760    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-5lwqk_70256eb4-eec2-4d39-8420-5403098c7cbf/kube-rbac-proxy/0.log"
Sep 17 17:51:37.108406 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.108373    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c9c9875f6-9pxqv_e7e2b070-60a7-4018-8457-1676ebd17592/telemeter-client/0.log"
Sep 17 17:51:37.137643 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.137617    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c9c9875f6-9pxqv_e7e2b070-60a7-4018-8457-1676ebd17592/reload/0.log"
Sep 17 17:51:37.165050 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.165026    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c9c9875f6-9pxqv_e7e2b070-60a7-4018-8457-1676ebd17592/kube-rbac-proxy/0.log"
Sep 17 17:51:37.454420 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.454349    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2krt9/must-gather-2pcjg" podStartSLOduration=11.611677148 podStartE2EDuration="12.454326647s" totalImagesPullingTime="842.649499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:26.910009185 +0000 UTC m=+4166.332947043" watchObservedRunningTime="2026-09-17 17:51:37.454326647 +0000 UTC m=+4176.877264523"
Sep 17 17:51:37.456644 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.456575    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"]
Sep 17 17:51:37.462268 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.462240    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.475460 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.475435    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"]
Sep 17 17:51:37.625093 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.625054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/01ba4b82-2355-493e-ad97-c34b37d9e170-proc\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.625270 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.625112    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6kg5\" (UniqueName: \"kubernetes.io/projected/01ba4b82-2355-493e-ad97-c34b37d9e170-kube-api-access-c6kg5\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.625270 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.625249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01ba4b82-2355-493e-ad97-c34b37d9e170-sys\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.625364 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.625308    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01ba4b82-2355-493e-ad97-c34b37d9e170-lib-modules\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.625364 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.625335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/01ba4b82-2355-493e-ad97-c34b37d9e170-podres\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.726756 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.726670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/01ba4b82-2355-493e-ad97-c34b37d9e170-proc\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.726756 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.726736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6kg5\" (UniqueName: \"kubernetes.io/projected/01ba4b82-2355-493e-ad97-c34b37d9e170-kube-api-access-c6kg5\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.726957 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.726806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/01ba4b82-2355-493e-ad97-c34b37d9e170-proc\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.726957 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.726876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01ba4b82-2355-493e-ad97-c34b37d9e170-sys\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.726957 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.726921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01ba4b82-2355-493e-ad97-c34b37d9e170-lib-modules\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.726957 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.726944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/01ba4b82-2355-493e-ad97-c34b37d9e170-podres\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.727150 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.727029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01ba4b82-2355-493e-ad97-c34b37d9e170-sys\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.727150 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.727126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01ba4b82-2355-493e-ad97-c34b37d9e170-lib-modules\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.727220 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.727129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/01ba4b82-2355-493e-ad97-c34b37d9e170-podres\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.737368 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.737336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6kg5\" (UniqueName: \"kubernetes.io/projected/01ba4b82-2355-493e-ad97-c34b37d9e170-kube-api-access-c6kg5\") pod \"perf-node-gather-daemonset-8tzjw\" (UID: \"01ba4b82-2355-493e-ad97-c34b37d9e170\") " pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:37.776948 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:37.776915    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:38.068148 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:38.068117    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"]
Sep 17 17:51:38.072698 ip-10-0-140-49 kubenswrapper[2581]: W0917 17:51:38.070784    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod01ba4b82_2355_493e_ad97_c34b37d9e170.slice/crio-5c07469d167ba861a659caf78286009faf3b032089c86f447cd9ca78386482aa WatchSource:0}: Error finding container 5c07469d167ba861a659caf78286009faf3b032089c86f447cd9ca78386482aa: Status 404 returned error can't find the container with id 5c07469d167ba861a659caf78286009faf3b032089c86f447cd9ca78386482aa
Sep 17 17:51:38.966037 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:38.965987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw" event={"ID":"01ba4b82-2355-493e-ad97-c34b37d9e170","Type":"ContainerStarted","Data":"d4ed601c6197202678aa6657eec4c8ddef840237f476ccfa382b116b1ba5ced9"}
Sep 17 17:51:38.966232 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:38.966052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw" event={"ID":"01ba4b82-2355-493e-ad97-c34b37d9e170","Type":"ContainerStarted","Data":"5c07469d167ba861a659caf78286009faf3b032089c86f447cd9ca78386482aa"}
Sep 17 17:51:38.966232 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:38.966167    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:38.993842 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:38.993788    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw" podStartSLOduration=1.993774729 podStartE2EDuration="1.993774729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:38.991829191 +0000 UTC m=+4178.414767055" watchObservedRunningTime="2026-09-17 17:51:38.993774729 +0000 UTC m=+4178.416712591"
Sep 17 17:51:39.493366 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:39.493329    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6745bf549f-srngd_6e76ee1e-ca4d-4af7-bf2d-8ea49d245f7d/console/0.log"
Sep 17 17:51:39.554192 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:39.554156    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-6h4np_4eb30ced-c505-47e5-aca7-abbe51e37473/download-server/0.log"
Sep 17 17:51:41.653089 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:41.653054    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gqqcv_c2fed88c-fffb-4c6f-9532-5af8fccb6b98/dns/0.log"
Sep 17 17:51:41.741166 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:41.741138    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gqqcv_c2fed88c-fffb-4c6f-9532-5af8fccb6b98/kube-rbac-proxy/0.log"
Sep 17 17:51:41.997960 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:41.997931    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rkpps_d0aa3902-ccf1-49cf-ad48-aafa74d040f0/dns-node-resolver/0.log"
Sep 17 17:51:42.719709 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:42.719679    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-555b8df8cf-7hc5c_4fe9c5b4-7dd2-4c5e-acac-531a4aecf301/registry/0.log"
Sep 17 17:51:42.967961 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:42.967926    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kfplg_0513c985-c365-41ee-959d-b02794d96aab/node-ca/0.log"
Sep 17 17:51:44.982988 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:44.982943    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2krt9/perf-node-gather-daemonset-8tzjw"
Sep 17 17:51:45.131488 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:45.131455    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l6gdl_328baa78-1822-4052-8b99-72fc548a3ece/serve-healthcheck-canary/0.log"
Sep 17 17:51:46.223198 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:46.223169    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mwc8d_a5182664-069c-4e58-acdc-327576520ea1/kube-rbac-proxy/0.log"
Sep 17 17:51:46.277818 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:46.277775    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mwc8d_a5182664-069c-4e58-acdc-327576520ea1/exporter/0.log"
Sep 17 17:51:46.323890 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:46.323852    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mwc8d_a5182664-069c-4e58-acdc-327576520ea1/extractor/0.log"
Sep 17 17:51:51.197116 ip-10-0-140-49 kubenswrapper[2581]: I0917 17:51:51.197083    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-b8d4866b4-vflkm_d08f318a-6fc4-42aa-a120-aad03695f9ac/manager/0.log"