Sep 17 15:36:48.914346 ip-10-0-129-37 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 15:36:48.914359 ip-10-0-129-37 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 15:36:48.914366 ip-10-0-129-37 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 15:36:48.914533 ip-10-0-129-37 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 15:36:59.163624 ip-10-0-129-37 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 15:36:59.163642 ip-10-0-129-37 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot db2663f37ca64023a35990e7de2d4a3b --
Sep 17 15:38:54.414770 ip-10-0-129-37 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 15:38:54.823482 ip-10-0-129-37 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:38:54.823482 ip-10-0-129-37 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 15:38:54.823482 ip-10-0-129-37 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:38:54.823482 ip-10-0-129-37 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 15:38:54.823482 ip-10-0-129-37 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:38:54.825437 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.825344    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 15:38:54.831398 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831383    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:54.831398 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831399    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831405    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831408    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831412    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831415    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831418    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831421    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831424    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831427    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831429    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831432    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831435    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831449    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831453    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831456    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831459    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831461    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831464    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831467    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831470    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:54.831482 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831473    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831476    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831479    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831481    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831484    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831487    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831490    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831493    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831496    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831499    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831502    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831504    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831507    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831509    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831512    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831515    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831519    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831523    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831526    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831529    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:54.832176 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831531    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831534    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831537    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831539    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831541    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831544    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831547    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831549    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831552    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831554    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831557    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831560    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831565    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831568    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831572    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831575    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831578    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831582    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831585    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:54.832824 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831587    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831858    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831869    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831874    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831879    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831883    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831888    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831892    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831897    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831905    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831911    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831916    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831927    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831932    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831937    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831941    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831946    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831951    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831970    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831975    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:54.833297 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831980    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831985    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831989    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.831994    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832004    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832008    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832015    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832022    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832035    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832039    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832044    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832048    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832053    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832057    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832062    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832066    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832075    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832079    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.832084    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833360    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:54.833795 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833371    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833375    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833379    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833382    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833385    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833388    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833392    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833396    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833399    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833402    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833405    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833408    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833410    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833413    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833415    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833418    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833421    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833423    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833426    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:54.834318 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833428    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833431    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833433    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833454    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833460    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833465    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833469    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833473    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833476    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833479    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833481    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833484    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833486    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833489    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833492    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833495    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833498    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833500    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833503    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:54.834807 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833506    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833508    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833511    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833513    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833516    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833519    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833521    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833524    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833526    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833530    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833532    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833535    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833538    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833540    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833543    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833545    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833548    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833550    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833553    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833557    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:54.835291 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833561    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833565    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833567    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833570    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833573    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833577    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833580    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833583    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833586    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833588    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833591    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833593    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833596    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833599    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833601    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833604    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833606    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833609    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833611    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833614    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:54.835782 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833617    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833619    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833622    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833629    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833632    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833634    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833637    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833639    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833642    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833645    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833647    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833650    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833652    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833655    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833657    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833660    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833662    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833666    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833668    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.833671    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:54.836289 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834050    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834055    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834059    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834062    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834065    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834068    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834071    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834075    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834079    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834082    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834085    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834089    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834091    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834094    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834097    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834100    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834102    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834104    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834107    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834109    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:54.836792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834112    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834115    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834117    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834120    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834122    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834124    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834127    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834130    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834132    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834136    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834139    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834142    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834144    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834147    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834149    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834152    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834154    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834157    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834159    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834162    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:54.837287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834164    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834167    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834171    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834174    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834178    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834181    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834183    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834186    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834188    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834190    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834193    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834195    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834198    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834200    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834203    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834206    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834208    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834210    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834213    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:54.837792 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834215    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834218    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834221    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834224    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834227    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834229    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834232    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834234    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834237    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834240    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834242    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834245    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834248    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834250    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834253    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834255    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834258    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834261    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834265    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834267    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:54.838250 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834270    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834272    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834275    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834277    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834280    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834282    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834285    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834288    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834290    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834293    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834295    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834298    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834300    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834303    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834307    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834311    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834314    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834317    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834320    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.834323    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:54.838752 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834392    2574 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834429    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834437    2574 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834457    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834462    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834465    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834469    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834474    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834477    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834480    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834484    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834488    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834492    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834495    2574 flags.go:64] FLAG: --cgroup-root=""
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834498    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834501    2574 flags.go:64] FLAG: --client-ca-file=""
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834504    2574 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834507    2574 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834511    2574 flags.go:64] FLAG: --cluster-domain=""
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834514    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834517    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834520    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834523    2574 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834527    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 15:38:54.839277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834530    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834534    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834537    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834540    2574 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834543    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834546    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834549    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834552    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834556    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834559    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834561    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834564    2574 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834567    2574 flags.go:64] FLAG: --enable-server="true"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834570    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834575    2574 flags.go:64] FLAG: --event-burst="100"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834578    2574 flags.go:64] FLAG: --event-qps="50"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834581    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834584    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834587    2574 flags.go:64] FLAG: --eviction-hard=""
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834591    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834594    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834598    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834601    2574 flags.go:64] FLAG: --eviction-soft=""
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834604    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834606    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 15:38:54.839884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834609    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834612    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834615    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834618    2574 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834621    2574 flags.go:64] FLAG: --feature-gates=""
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834624    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834628    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834631    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834637    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834640    2574 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834643    2574 flags.go:64] FLAG: --help="false"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834646    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-129-37.ec2.internal"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834650    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834653    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834656    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834659    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834663    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834665    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834668    2574 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834671    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834675    2574 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834677    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834681    2574 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834684    2574 flags.go:64] FLAG: --kube-reserved=""
Sep 17 15:38:54.840525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834686    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834689    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834692    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834695    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834699    2574 flags.go:64] FLAG: --lock-file=""
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834702    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834705    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834708    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834713    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834716    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834719    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834722    2574 flags.go:64] FLAG: --logging-format="text"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834725    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834728    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834731    2574 flags.go:64] FLAG: --manifest-url=""
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834734    2574 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834740    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834743    2574 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834747    2574 flags.go:64] FLAG: --max-pods="110"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834750    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834753    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834756    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834759    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834762    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834765    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 15:38:54.841105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834768    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834775    2574 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834779    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834781    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834784    2574 flags.go:64] FLAG: --pod-cidr=""
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834787    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834793    2574 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834796    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834799    2574 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834801    2574 flags.go:64] FLAG: --port="10250"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834805    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834808    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0af4142e003f9a5fb"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834812    2574 flags.go:64] FLAG: --qos-reserved=""
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834815    2574 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834820    2574 flags.go:64] FLAG: --register-node="true"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834823    2574 flags.go:64] FLAG: --register-with-taints=""
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834826    2574 flags.go:64] FLAG: --registry-burst="10"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834829    2574 flags.go:64] FLAG: --registry-qps="5"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834832    2574 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834835    2574 flags.go:64] FLAG: --reserved-memory=""
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834839    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834842    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834845    2574 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834848    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834853    2574 flags.go:64] FLAG: --runonce="false"
Sep 17 15:38:54.841735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834856    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834859    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834862    2574 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834864    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834867    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834870    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834873    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834876    2574 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834879    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834882    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834885    2574 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834887    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834890    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834893    2574 flags.go:64] FLAG: --system-cgroups=""
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834896    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834901    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834904    2574 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834907    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834911    2574 flags.go:64] FLAG: --tls-min-version=""
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834914    2574 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834926    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834931    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834934    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834937    2574 flags.go:64] FLAG: --v="2"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834944    2574 flags.go:64] FLAG: --version="false"
Sep 17 15:38:54.842356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834948    2574 flags.go:64] FLAG: --vmodule=""
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834952    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.834955    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835071    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835075    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835080    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835086    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835088    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835091    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835094    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835096    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835099    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835102    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835104    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835107    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835110    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835112    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835115    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835119    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835122    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:54.843019 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835126    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835129    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835132    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835134    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835138    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835140    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835143    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835146    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835151    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835153    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835156    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835159    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835161    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835164    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835167    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835169    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835172    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835175    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835180    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835183    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:54.843527 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835186    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835188    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835191    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835193    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835196    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835199    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835201    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835204    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835207    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835210    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835212    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835215    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835217    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835220    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835222    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835225    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835227    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835230    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835233    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:54.844111 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835235    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835239    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835242    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835245    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835247    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835249    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835252    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835255    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835257    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835260    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835263    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835267    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835269    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835272    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835275    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835277    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835280    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835282    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835285    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835288    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:54.844602 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835290    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835293    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835296    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835299    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835302    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835304    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835307    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835310    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835312    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835315    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835317    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835320    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835323    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835327    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835331    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835335    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835339    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835341    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835344    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835347    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:54.845092 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835350    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:54.845590 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835353    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:54.845590 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.835355    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:54.845590 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.836130    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:38:54.845590 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.843165    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 15:38:54.845590 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.843182    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 15:38:54.845590 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.844274    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 15:38:54.845590 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.844281    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 15:38:54.845590 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844338    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:54.845590 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844344    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:54.845590 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844347    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:54.845590 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844350    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:54.845590 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844353    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:54.845590 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844356    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:54.845590 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844361    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844366    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844369    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844372    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844374    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844377    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844379    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844383    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844386    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844388    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844391    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844394    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844396    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844399    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844401    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844404    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844407    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844409    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844412    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:54.845983 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844415    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844419    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844423    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844426    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844431    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844434    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844447    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844450    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844453    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844455    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844473    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844476    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844480    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844483    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844486    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844488    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844491    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844494    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844498    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844501    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:54.846461 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844504    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844507    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844509    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844512    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844515    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844517    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844520    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844523    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844525    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844528    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844530    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844533    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844535    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844538    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844540    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844544    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844549    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844553    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844556    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:54.846951 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844559    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844561    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844564    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844567    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844570    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844572    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844575    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844578    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844580    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844583    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844586    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844589    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844591    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844594    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844596    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844600    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844602    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844605    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844608    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844610    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:54.847425 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844613    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844616    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844619    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844621    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844624    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844626    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844629    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844632    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844634    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844638    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844641    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844644    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844647    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844649    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844652    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:54.847919 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.844657    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844781    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844786    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844802    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844806    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844809    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844812    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844815    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844818    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844821    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844824    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844827    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844830    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844833    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844836    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844840    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844842    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844845    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844848    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844850    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:54.848287 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844853    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844856    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844858    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844861    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844863    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844866    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844869    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844874    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844877    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844881    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844884    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844887    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844890    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844892    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844895    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844898    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844900    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844903    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844906    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844908    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:54.848790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844911    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844913    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844917    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844920    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844924    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844927    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844930    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844933    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844936    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844938    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844941    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844944    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844946    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844949    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844951    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844954    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844957    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844960    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844963    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:54.849269 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844966    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844970    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844973    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844977    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844979    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844982    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844984    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844987    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844990    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844993    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844995    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.844998    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845000    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845003    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845005    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845008    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845010    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845013    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845015    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845018    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:54.849738 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845021    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845023    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845025    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845028    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845031    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845033    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845036    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845038    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845041    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845043    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845046    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845049    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845052    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845054    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845057    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845060    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845063    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845065    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845068    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845070    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:54.850215 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:54.845073    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:54.850697 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.845078    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:38:54.850697 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.845198    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 15:38:54.850697 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.847746    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 15:38:54.850697 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.848635    2574 server.go:1019] "Starting client certificate rotation"
Sep 17 15:38:54.850697 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.848736    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:38:54.850697 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.848777    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:38:54.851760 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.851746    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:38:54.852872 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.852855    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:38:54.869201 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.867779    2574 log.go:25] "Validated CRI v1 runtime API"
Sep 17 15:38:54.871883 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.871866    2574 log.go:25] "Validated CRI v1 image API"
Sep 17 15:38:54.872740 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.872722    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 15:38:54.876002 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.875983    2574 fs.go:135] Filesystem UUIDs: map[1e610d0d-bb30-47ce-bd65-60f7eb28ecdb:/dev/nvme0n1p4 6dc257cb-2d81-4c4d-a5a0-d58170fce517:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 17 15:38:54.876050 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.876002    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 15:38:54.882707 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.882595    2574 manager.go:216] Machine: {Timestamp:2026-09-17 15:38:54.879613076 +0000 UTC m=+0.351452913 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3138621 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea3629a09de1cc228e65e8067635a SystemUUID:ec2ea362-9a09-de1c-c228-e65e8067635a BootID:db2663f3-7ca6-4023-a359-90e7de2d4a3b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:da:9f:3d:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:da:9f:3d:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:97:b0:be:62:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 15:38:54.882707 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.882704    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 15:38:54.882806 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.882790    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 15:38:54.884416 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.884308    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:38:54.886258 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.886227    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 15:38:54.886397 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.886262    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-37.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 15:38:54.886987 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.886975    2574 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 15:38:54.886987 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.886987    2574 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 15:38:54.887101 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.886995    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:38:54.887601 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.887591    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:38:54.888384 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.888373    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 15:38:54.893219 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.893207    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:38:54.893340 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.893331    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 15:38:54.893384 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.893361    2574 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 15:38:54.893384 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.893371    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 15:38:54.893494 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.893388    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 15:38:54.893494 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.893398    2574 kubelet.go:398] "Adding apiserver pod source"
Sep 17 15:38:54.893494 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.893408    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 15:38:54.893815 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.893804    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:38:54.893851 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.893826    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:38:54.895578 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.895544    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 15:38:54.897763 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.897750    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 15:38:54.897810 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.897780    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 15:38:54.899372 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.899360    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 15:38:54.899414 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.899377    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 15:38:54.899414 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.899382    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 15:38:54.899414 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.899388    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 15:38:54.899414 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.899397    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 15:38:54.899414 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.899402    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 15:38:54.899414 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.899416    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 15:38:54.899601 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.899421    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 15:38:54.899601 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.899428    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 15:38:54.899601 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.899433    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 15:38:54.899601 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.899452    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 15:38:54.899601 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.899457    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 15:38:54.900956 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.900942    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 15:38:54.900989 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.900959    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 15:38:54.905118 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.905105    2574 server.go:1268] "Started kubelet"
Sep 17 15:38:54.905263 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.905239    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 15:38:54.905319 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:54.905101    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 15:38:54.905319 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.905292    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 15:38:54.905414 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:54.905378    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 15:38:54.905414 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.905375    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 15:38:54.905527 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.905514    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-37.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 15:38:54.905863 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.905844    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bskfz"
Sep 17 15:38:54.905985 ip-10-0-129-37 systemd[1]: Started Kubernetes Kubelet.
Sep 17 15:38:54.906308 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.906293    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 15:38:54.910897 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.910876    2574 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 15:38:54.911468 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:54.911433    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 15:38:54.912971 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.912954    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bskfz"
Sep 17 15:38:54.914173 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:54.913241    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-37.ec2.internal.18d625dd6ee87d3f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-37.ec2.internal,UID:ip-10-0-129-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-37.ec2.internal,},FirstTimestamp:2026-09-17 15:38:54.905081151 +0000 UTC m=+0.376920990,LastTimestamp:2026-09-17 15:38:54.905081151 +0000 UTC m=+0.376920990,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-37.ec2.internal,}"
Sep 17 15:38:54.915927 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.915909    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 15:38:54.916883 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.916868    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 15:38:54.917008 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:54.916987    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 17 15:38:54.917661 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.917644    2574 factory.go:55] Registering systemd factory
Sep 17 15:38:54.917758 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.917665    2574 factory.go:223] Registration of the systemd container factory successfully
Sep 17 15:38:54.917758 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.917712    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 15:38:54.917758 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.917729    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 15:38:54.917758 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.917713    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 15:38:54.917976 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.917839    2574 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 15:38:54.917976 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.917847    2574 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 15:38:54.917976 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.917903    2574 factory.go:153] Registering CRI-O factory
Sep 17 15:38:54.917976 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.917921    2574 factory.go:223] Registration of the crio container factory successfully
Sep 17 15:38:54.918094 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.917993    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 15:38:54.918094 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.918012    2574 factory.go:103] Registering Raw factory
Sep 17 15:38:54.918094 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.918028    2574 manager.go:1196] Started watching for new ooms in manager
Sep 17 15:38:54.918419 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.918406    2574 manager.go:349] Starting recovery of all containers
Sep 17 15:38:54.925420 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.925401    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:54.925984 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.925966    2574 manager.go:354] Recovery completed
Sep 17 15:38:54.932740 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.932631    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:54.933040 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:54.933025    2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-37.ec2.internal\" not found" node="ip-10-0-129-37.ec2.internal"
Sep 17 15:38:54.934526 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.934508    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:54.934617 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.934539    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:54.934617 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.934550    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:54.935203 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.935188    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:54.935341 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.935180    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 15:38:54.935341 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.935233    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 15:38:54.935341 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.935253    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:38:54.935979 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.935964    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:54.936052 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.935994    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:54.936052 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.936016    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:54.937467 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.937454    2574 policy_none.go:49] "None policy: Start"
Sep 17 15:38:54.937531 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.937472    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 15:38:54.937531 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.937486    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 15:38:54.939511 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:54.939499    2574 policy_none.go:47] "Start"
Sep 17 15:38:55.018127 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.018080    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 17 15:38:55.040813 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.040773    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 15:38:55.042065 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.042048    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 15:38:55.042151 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.042109    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 15:38:55.042151 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.042135    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 15:38:55.042254 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.042237    2574 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 17 15:38:55.045016 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.044999    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:55.111824 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.111809    2574 manager.go:337] "Starting Device Plugin manager"
Sep 17 15:38:55.111898 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.111851    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 15:38:55.111898 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.111861    2574 server.go:87] "Starting device plugin registration server"
Sep 17 15:38:55.112255 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.112242    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 15:38:55.112327 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.112259    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 15:38:55.112381 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.112337    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 15:38:55.112462 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.112432    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 15:38:55.112462 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.112462    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 15:38:55.113025 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.113006    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 15:38:55.113106 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.113061    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 17 15:38:55.142352 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.142328    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"]
Sep 17 15:38:55.142421 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.142405    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:55.144247 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.144233    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:55.144323 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.144261    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:55.144323 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.144274    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:55.146693 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.146680    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:55.146823 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.146810    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.146856 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.146839    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:55.147465 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.147436    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:55.147531 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.147478    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:55.147531 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.147494    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:55.147589 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.147495    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:55.147589 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.147574    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:55.147589 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.147589    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:55.149800 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.149785    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.149885 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.149811    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:55.150574 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.150556    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:55.150657 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.150584    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:55.150657 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.150594    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:55.181179 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.181159    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-37.ec2.internal\" not found" node="ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.185586 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.185569    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-37.ec2.internal\" not found" node="ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.212557 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.212532    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:55.215135 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.215118    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:55.215241 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.215194    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:55.215241 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.215206    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:55.215241 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.215229    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.218992 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.218973    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5723e815bf040e5d45b5f45f3fe46312-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal\" (UID: \"5723e815bf040e5d45b5f45f3fe46312\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.219078 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.218997    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5723e815bf040e5d45b5f45f3fe46312-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal\" (UID: \"5723e815bf040e5d45b5f45f3fe46312\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.219078 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.219017    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6abc4d8c8ce3d693c97b0f59444f7f92-config\") pod \"kube-apiserver-proxy-ip-10-0-129-37.ec2.internal\" (UID: \"6abc4d8c8ce3d693c97b0f59444f7f92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.224736 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.224718    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.224802 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.224737    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-37.ec2.internal\": node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 17 15:38:55.239422 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.239397    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 17 15:38:55.320179 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.320139    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5723e815bf040e5d45b5f45f3fe46312-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal\" (UID: \"5723e815bf040e5d45b5f45f3fe46312\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.320179 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.320182    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5723e815bf040e5d45b5f45f3fe46312-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal\" (UID: \"5723e815bf040e5d45b5f45f3fe46312\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.320390 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.320198    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6abc4d8c8ce3d693c97b0f59444f7f92-config\") pod \"kube-apiserver-proxy-ip-10-0-129-37.ec2.internal\" (UID: \"6abc4d8c8ce3d693c97b0f59444f7f92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.320390 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.320267    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5723e815bf040e5d45b5f45f3fe46312-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal\" (UID: \"5723e815bf040e5d45b5f45f3fe46312\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.320390 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.320278    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6abc4d8c8ce3d693c97b0f59444f7f92-config\") pod \"kube-apiserver-proxy-ip-10-0-129-37.ec2.internal\" (UID: \"6abc4d8c8ce3d693c97b0f59444f7f92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.320390 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.320268    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5723e815bf040e5d45b5f45f3fe46312-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal\" (UID: \"5723e815bf040e5d45b5f45f3fe46312\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.340309 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.340240    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 17 15:38:55.441507 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.441455    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 17 15:38:55.485240 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.485201    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.488393 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.488377    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:55.542616 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.542577    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 17 15:38:55.643684 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.643581    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 17 15:38:55.745126 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.745087    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 17 15:38:55.845977 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.845943    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 17 15:38:55.848178 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.848157    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 15:38:55.848360 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.848336    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:38:55.848424 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.848347    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:38:55.919089 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.918992    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 15:33:55 +0000 UTC" deadline="2028-02-26 20:34:12.864090901 +0000 UTC"
Sep 17 15:38:55.919089 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:55.919023    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12652h55m16.945072598s"
Sep 17 15:38:55.947040 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:55.947003    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 17 15:38:56.047550 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:56.047512    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 17 15:38:56.139454 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.139419    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:56.176809 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:56.176773    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abc4d8c8ce3d693c97b0f59444f7f92.slice/crio-1b6eff5492280210b7f89f6e48021f39da5184f9037d9c524db383410a059ceb WatchSource:0}: Error finding container 1b6eff5492280210b7f89f6e48021f39da5184f9037d9c524db383410a059ceb: Status 404 returned error can't find the container with id 1b6eff5492280210b7f89f6e48021f39da5184f9037d9c524db383410a059ceb
Sep 17 15:38:56.177203 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:56.177179    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5723e815bf040e5d45b5f45f3fe46312.slice/crio-1eeec627f2d404cd3180cc75fc6241c18cf5a996190fbbebce93ddb708b99e3d WatchSource:0}: Error finding container 1eeec627f2d404cd3180cc75fc6241c18cf5a996190fbbebce93ddb708b99e3d: Status 404 returned error can't find the container with id 1eeec627f2d404cd3180cc75fc6241c18cf5a996190fbbebce93ddb708b99e3d
Sep 17 15:38:56.182603 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.182585    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:38:56.189686 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.189667    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:56.217588 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.217555    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:56.229469 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.229437    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:38:56.230395 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.230381    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 17 15:38:56.242506 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.242481    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:38:56.678486 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.678429    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:56.894705 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.894669    2574 apiserver.go:52] "Watching apiserver"
Sep 17 15:38:56.901356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.901327    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 15:38:56.902142 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.902109    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9","openshift-cluster-node-tuning-operator/tuned-qr6th","openshift-image-registry/node-ca-7845q","openshift-multus/multus-bvgdz","openshift-multus/network-metrics-daemon-c4llb","kube-system/global-pull-secret-syncer-vt2l8","kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal","openshift-dns/node-resolver-vzk9l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal","openshift-multus/multus-additional-cni-plugins-njl24","openshift-network-diagnostics/network-check-target-twzfl","openshift-network-operator/iptables-alerter-k8lsf","openshift-ovn-kubernetes/ovnkube-node-hwfgx","kube-system/konnectivity-agent-9t4xp"]
Sep 17 15:38:56.904968 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.904940    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vzk9l"
Sep 17 15:38:56.907314 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.907172    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-brk72\""
Sep 17 15:38:56.907422 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.907383    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 15:38:56.907422 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.907416    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 15:38:56.907616 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.907600    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.910219 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.909769    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:38:56.910219 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.910194    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7845q"
Sep 17 15:38:56.911551 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.911531    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 15:38:56.911835 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.911771    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bdc4r\""
Sep 17 15:38:56.912751 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.912406    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 15:38:56.912751 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.912702    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-74mqt\""
Sep 17 15:38:56.913082 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.913063    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 15:38:56.913364 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.913346    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 15:38:56.913693 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.913675    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:56.915953 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.915532    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 15:38:56.916885 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.916104    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k8lsf"
Sep 17 15:38:56.916885 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.916323    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 15:38:56.916885 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.916588    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 15:38:56.916885 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.916669    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qt7tp\""
Sep 17 15:38:56.916885 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.916839    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 15:38:56.918612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.917837    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:56.920635 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.920208    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 15:38:56.920635 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.920471    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-69wmc\""
Sep 17 15:38:56.921411 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.920817    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.921411 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.921049    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:38:56.921411 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.921317    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 15:38:56.921657 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.921570    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 15:38:56.921657 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.921593    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 15:38:56.921768 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.921669    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 15:38:56.921768 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.921575    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v2ldn\""
Sep 17 15:38:56.921870 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.921701    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 15:38:56.921870 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.921792    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 15:38:56.922753 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.922580    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 15:38:56.923066 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.922878    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:38:56.923066 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:56.922967    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:38:56.924346 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.924159    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wt2hf\""
Sep 17 15:38:56.925631 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.925612    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:38:56.925975 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:56.925952    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:38:56.929740 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.929611    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:38:56.929740 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:56.929709    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:38:56.929888 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.929838    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-sys\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.929954 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.929868    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:56.929954 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.929916    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-socket-dir\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:56.929954 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.929942    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj79l\" (UniqueName: \"kubernetes.io/projected/34864426-aeff-4ea6-8bc1-145d6ee3f791-kube-api-access-nj79l\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:56.930135 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.929970    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-system-cni-dir\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.930135 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.929995    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-var-lib-cni-multus\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.930135 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930024    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc3bbb2e-6060-4eca-89d0-33cc860b2039-tmp\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.930135 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930046    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/346c8e4c-4119-41a8-b798-c23d6de9b203-serviceca\") pod \"node-ca-7845q\" (UID: \"346c8e4c-4119-41a8-b798-c23d6de9b203\") " pod="openshift-image-registry/node-ca-7845q"
Sep 17 15:38:56.930135 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930074    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6588c63a-cfe8-415b-a31f-aecc29f7f69c-cni-binary-copy\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:56.930135 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930098    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-cni-binary-copy\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.930470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930141    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-run-k8s-cni-cncf-io\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.930470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930167    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-run\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.930470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930192    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92w8w\" (UniqueName: \"kubernetes.io/projected/cc3bbb2e-6060-4eca-89d0-33cc860b2039-kube-api-access-92w8w\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.930470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930218    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm8gk\" (UniqueName: \"kubernetes.io/projected/cb06d52b-3ebb-4d34-a72e-c4a109808a4a-kube-api-access-vm8gk\") pod \"iptables-alerter-k8lsf\" (UID: \"cb06d52b-3ebb-4d34-a72e-c4a109808a4a\") " pod="openshift-network-operator/iptables-alerter-k8lsf"
Sep 17 15:38:56.930470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930243    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8f3b74-61b5-4b22-bf59-ab2ec9054df1-tmp-dir\") pod \"node-resolver-vzk9l\" (UID: \"9f8f3b74-61b5-4b22-bf59-ab2ec9054df1\") " pod="openshift-dns/node-resolver-vzk9l"
Sep 17 15:38:56.930470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930268    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6588c63a-cfe8-415b-a31f-aecc29f7f69c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:56.930470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-multus-cni-dir\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.930470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930320    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-sysctl-d\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.930470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930345    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-sysctl-conf\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.930470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930370    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-device-dir\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:56.930470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930395    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-sys-fs\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:56.930470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930419    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb06d52b-3ebb-4d34-a72e-c4a109808a4a-host-slash\") pod \"iptables-alerter-k8lsf\" (UID: \"cb06d52b-3ebb-4d34-a72e-c4a109808a4a\") " pod="openshift-network-operator/iptables-alerter-k8lsf"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930459    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw6xk\" (UniqueName: \"kubernetes.io/projected/9f8f3b74-61b5-4b22-bf59-ab2ec9054df1-kube-api-access-kw6xk\") pod \"node-resolver-vzk9l\" (UID: \"9f8f3b74-61b5-4b22-bf59-ab2ec9054df1\") " pod="openshift-dns/node-resolver-vzk9l"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930517    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzx86\" (UniqueName: \"kubernetes.io/projected/6588c63a-cfe8-415b-a31f-aecc29f7f69c-kube-api-access-dzx86\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930556    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-etc-kubernetes\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930595    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-modprobe-d\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930649    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-var-lib-kubelet\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930676    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-host\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930702    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930727    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-cnibin\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930752    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-run-netns\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930775    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-hostroot\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930799    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-sysconfig\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930822    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-lib-modules\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930850    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7wpq\" (UniqueName: \"kubernetes.io/projected/346c8e4c-4119-41a8-b798-c23d6de9b203-kube-api-access-r7wpq\") pod \"node-ca-7845q\" (UID: \"346c8e4c-4119-41a8-b798-c23d6de9b203\") " pod="openshift-image-registry/node-ca-7845q"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930877    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-registration-dir\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930903    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb06d52b-3ebb-4d34-a72e-c4a109808a4a-iptables-alerter-script\") pod \"iptables-alerter-k8lsf\" (UID: \"cb06d52b-3ebb-4d34-a72e-c4a109808a4a\") " pod="openshift-network-operator/iptables-alerter-k8lsf"
Sep 17 15:38:56.932433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930951    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-var-lib-cni-bin\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.930976    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-var-lib-kubelet\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931001    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-run-multus-certs\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931043    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6588c63a-cfe8-415b-a31f-aecc29f7f69c-system-cni-dir\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6588c63a-cfe8-415b-a31f-aecc29f7f69c-cnibin\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931101    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6588c63a-cfe8-415b-a31f-aecc29f7f69c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931129    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-multus-socket-dir-parent\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931165    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f8f3b74-61b5-4b22-bf59-ab2ec9054df1-hosts-file\") pod \"node-resolver-vzk9l\" (UID: \"9f8f3b74-61b5-4b22-bf59-ab2ec9054df1\") " pod="openshift-dns/node-resolver-vzk9l"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-multus-conf-dir\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931296    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ntrq\" (UniqueName: \"kubernetes.io/projected/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-kube-api-access-7ntrq\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931324    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-kubernetes\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931348    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-systemd\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931372    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-tuned\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931397    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/346c8e4c-4119-41a8-b798-c23d6de9b203-host\") pod \"node-ca-7845q\" (UID: \"346c8e4c-4119-41a8-b798-c23d6de9b203\") " pod="openshift-image-registry/node-ca-7845q"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931432    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6588c63a-cfe8-415b-a31f-aecc29f7f69c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931475    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-multus-daemon-config\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.931622    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.932990    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:38:56.933282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.933125    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9t4xp"
Sep 17 15:38:56.934474 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.933734    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 15:38:56.934474 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.933753    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 15:38:56.936714 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.936681    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 15:38:56.937094 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.936981    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 15:38:56.937317 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.937253    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m8hjl\""
Sep 17 15:38:56.937800 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.937781    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 15:38:56.938580 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.938560    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lv52q\""
Sep 17 15:38:56.938841 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.938825    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 15:38:56.939649 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.939628    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 15:38:56.941802 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.941636    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 15:38:56.957631 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.957223    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h6h6w"
Sep 17 15:38:56.967431 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:56.966957    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h6h6w"
Sep 17 15:38:57.019430 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.019401    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 15:38:57.032082 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032050    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6588c63a-cfe8-415b-a31f-aecc29f7f69c-cni-binary-copy\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.032238 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032094    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-cni-binary-copy\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.032238 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032114    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-run\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.032238 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032132    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm8gk\" (UniqueName: \"kubernetes.io/projected/cb06d52b-3ebb-4d34-a72e-c4a109808a4a-kube-api-access-vm8gk\") pod \"iptables-alerter-k8lsf\" (UID: \"cb06d52b-3ebb-4d34-a72e-c4a109808a4a\") " pod="openshift-network-operator/iptables-alerter-k8lsf"
Sep 17 15:38:57.032238 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032153    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.032238 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032183    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8f3b74-61b5-4b22-bf59-ab2ec9054df1-tmp-dir\") pod \"node-resolver-vzk9l\" (UID: \"9f8f3b74-61b5-4b22-bf59-ab2ec9054df1\") " pod="openshift-dns/node-resolver-vzk9l"
Sep 17 15:38:57.032238 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-multus-cni-dir\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.032238 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032235    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-sys-fs\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032261    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzx86\" (UniqueName: \"kubernetes.io/projected/6588c63a-cfe8-415b-a31f-aecc29f7f69c-kube-api-access-dzx86\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032304    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-modprobe-d\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032336    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-host\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032364    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c35b1dbc-5400-46a6-a74a-1fac21cf6cff-konnectivity-ca\") pod \"konnectivity-agent-9t4xp\" (UID: \"c35b1dbc-5400-46a6-a74a-1fac21cf6cff\") " pod="kube-system/konnectivity-agent-9t4xp"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032362    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-multus-cni-dir\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032394    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-kubelet\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032421    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-cnibin\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032432    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-run\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032448    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-run-netns\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032489    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-sysconfig\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032526    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7wpq\" (UniqueName: \"kubernetes.io/projected/346c8e4c-4119-41a8-b798-c23d6de9b203-kube-api-access-r7wpq\") pod \"node-ca-7845q\" (UID: \"346c8e4c-4119-41a8-b798-c23d6de9b203\") " pod="openshift-image-registry/node-ca-7845q"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032551    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-registration-dir\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032569    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-modprobe-d\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032578    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb06d52b-3ebb-4d34-a72e-c4a109808a4a-iptables-alerter-script\") pod \"iptables-alerter-k8lsf\" (UID: \"cb06d52b-3ebb-4d34-a72e-c4a109808a4a\") " pod="openshift-network-operator/iptables-alerter-k8lsf"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032609    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs\") pod \"network-metrics-daemon-c4llb\" (UID: \"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf\") " pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032617    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-host\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.032732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032625    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8f3b74-61b5-4b22-bf59-ab2ec9054df1-tmp-dir\") pod \"node-resolver-vzk9l\" (UID: \"9f8f3b74-61b5-4b22-bf59-ab2ec9054df1\") " pod="openshift-dns/node-resolver-vzk9l"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032635    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-node-log\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-var-lib-cni-bin\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032673    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-sysconfig\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032695    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-var-lib-kubelet\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032723    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-run-multus-certs\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032725    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6588c63a-cfe8-415b-a31f-aecc29f7f69c-cni-binary-copy\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032750    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-run-netns\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032752    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6588c63a-cfe8-415b-a31f-aecc29f7f69c-cnibin\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032786    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-kubernetes\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032805    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-systemd\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032818    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-var-lib-cni-bin\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-run-systemd\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032729    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-cnibin\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032815    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-cni-binary-copy\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032857    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f8f3b74-61b5-4b22-bf59-ab2ec9054df1-hosts-file\") pod \"node-resolver-vzk9l\" (UID: \"9f8f3b74-61b5-4b22-bf59-ab2ec9054df1\") " pod="openshift-dns/node-resolver-vzk9l"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032870    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-systemd\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032723    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-sys-fs\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.033602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032886    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-registration-dir\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032938    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6588c63a-cfe8-415b-a31f-aecc29f7f69c-cnibin\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.032974    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-multus-conf-dir\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033003    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ntrq\" (UniqueName: \"kubernetes.io/projected/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-kube-api-access-7ntrq\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033021    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-kubernetes\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033031    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-tuned\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033043    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-multus-conf-dir\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033062    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f8f3b74-61b5-4b22-bf59-ab2ec9054df1-hosts-file\") pod \"node-resolver-vzk9l\" (UID: \"9f8f3b74-61b5-4b22-bf59-ab2ec9054df1\") " pod="openshift-dns/node-resolver-vzk9l"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033065    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033108    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033115    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-socket-dir\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033222    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-socket-dir\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033235    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-var-lib-kubelet\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033279    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj79l\" (UniqueName: \"kubernetes.io/projected/34864426-aeff-4ea6-8bc1-145d6ee3f791-kube-api-access-nj79l\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033302    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb06d52b-3ebb-4d34-a72e-c4a109808a4a-iptables-alerter-script\") pod \"iptables-alerter-k8lsf\" (UID: \"cb06d52b-3ebb-4d34-a72e-c4a109808a4a\") " pod="openshift-network-operator/iptables-alerter-k8lsf"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033343    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-run-multus-certs\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033311    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6588c63a-cfe8-415b-a31f-aecc29f7f69c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.034533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033387    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72f72846-2cc4-4793-a883-42382a787e30-kubelet-config\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033414    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr\") pod \"network-check-target-twzfl\" (UID: \"dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7\") " pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033432    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-slash\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-var-lib-cni-multus\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033506    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/346c8e4c-4119-41a8-b798-c23d6de9b203-serviceca\") pod \"node-ca-7845q\" (UID: \"346c8e4c-4119-41a8-b798-c23d6de9b203\") " pod="openshift-image-registry/node-ca-7845q"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033512    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6588c63a-cfe8-415b-a31f-aecc29f7f69c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033534    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kvmc\" (UniqueName: \"kubernetes.io/projected/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-kube-api-access-5kvmc\") pod \"network-metrics-daemon-c4llb\" (UID: \"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf\") " pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033560    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-run-openvswitch\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033507    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-var-lib-cni-multus\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033591    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-ovnkube-script-lib\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033569    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-run-k8s-cni-cncf-io\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033656    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92w8w\" (UniqueName: \"kubernetes.io/projected/cc3bbb2e-6060-4eca-89d0-33cc860b2039-kube-api-access-92w8w\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033699    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-host-run-k8s-cni-cncf-io\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033745    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-run-ovn\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033858    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033893    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/346c8e4c-4119-41a8-b798-c23d6de9b203-serviceca\") pod \"node-ca-7845q\" (UID: \"346c8e4c-4119-41a8-b798-c23d6de9b203\") " pod="openshift-image-registry/node-ca-7845q"
Sep 17 15:38:57.035609 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6588c63a-cfe8-415b-a31f-aecc29f7f69c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.033933    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-sysctl-d\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034029    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-sysctl-conf\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034067    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-device-dir\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034090    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb06d52b-3ebb-4d34-a72e-c4a109808a4a-host-slash\") pod \"iptables-alerter-k8lsf\" (UID: \"cb06d52b-3ebb-4d34-a72e-c4a109808a4a\") " pod="openshift-network-operator/iptables-alerter-k8lsf"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034112    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-sysctl-d\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034116    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw6xk\" (UniqueName: \"kubernetes.io/projected/9f8f3b74-61b5-4b22-bf59-ab2ec9054df1-kube-api-access-kw6xk\") pod \"node-resolver-vzk9l\" (UID: \"9f8f3b74-61b5-4b22-bf59-ab2ec9054df1\") " pod="openshift-dns/node-resolver-vzk9l"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034207    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-sysctl-conf\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034208    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-device-dir\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034240    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb06d52b-3ebb-4d34-a72e-c4a109808a4a-host-slash\") pod \"iptables-alerter-k8lsf\" (UID: \"cb06d52b-3ebb-4d34-a72e-c4a109808a4a\") " pod="openshift-network-operator/iptables-alerter-k8lsf"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034260    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-etc-kubernetes\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034292    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-var-lib-kubelet\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034314    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-etc-kubernetes\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034319    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034348    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72f72846-2cc4-4793-a883-42382a787e30-dbus\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034370    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-var-lib-kubelet\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034376    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:38:57.036612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034402    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34864426-aeff-4ea6-8bc1-145d6ee3f791-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034404    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-ovn-node-metrics-cert\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034449    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-hostroot\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034499    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6588c63a-cfe8-415b-a31f-aecc29f7f69c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-lib-modules\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-cni-bin\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034558    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-hostroot\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6588c63a-cfe8-415b-a31f-aecc29f7f69c-system-cni-dir\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034608    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6588c63a-cfe8-415b-a31f-aecc29f7f69c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034631    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-multus-socket-dir-parent\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034647    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6588c63a-cfe8-415b-a31f-aecc29f7f69c-system-cni-dir\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034674    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-etc-openvswitch\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034698    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-env-overrides\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034703    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-multus-socket-dir-parent\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034723    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/346c8e4c-4119-41a8-b798-c23d6de9b203-host\") pod \"node-ca-7845q\" (UID: \"346c8e4c-4119-41a8-b798-c23d6de9b203\") " pod="openshift-image-registry/node-ca-7845q"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034738    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-lib-modules\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034748    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-systemd-units\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.037666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034769    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-multus-daemon-config\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034773    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/346c8e4c-4119-41a8-b798-c23d6de9b203-host\") pod \"node-ca-7845q\" (UID: \"346c8e4c-4119-41a8-b798-c23d6de9b203\") " pod="openshift-image-registry/node-ca-7845q"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034783    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-sys\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034800    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c35b1dbc-5400-46a6-a74a-1fac21cf6cff-agent-certs\") pod \"konnectivity-agent-9t4xp\" (UID: \"c35b1dbc-5400-46a6-a74a-1fac21cf6cff\") " pod="kube-system/konnectivity-agent-9t4xp"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034815    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-run-netns\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034835    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-log-socket\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034858    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-ovnkube-config\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7gs4\" (UniqueName: \"kubernetes.io/projected/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-kube-api-access-j7gs4\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034910    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-system-cni-dir\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034923    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc3bbb2e-6060-4eca-89d0-33cc860b2039-sys\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc3bbb2e-6060-4eca-89d0-33cc860b2039-tmp\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-var-lib-openvswitch\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034989    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-cni-netd\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.034992    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-system-cni-dir\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.035216    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-multus-daemon-config\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.038643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.035274    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6588c63a-cfe8-415b-a31f-aecc29f7f69c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.039494 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.038681    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc3bbb2e-6060-4eca-89d0-33cc860b2039-tmp\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.039494 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.038728    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cc3bbb2e-6060-4eca-89d0-33cc860b2039-etc-tuned\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.048508 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.048441    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7wpq\" (UniqueName: \"kubernetes.io/projected/346c8e4c-4119-41a8-b798-c23d6de9b203-kube-api-access-r7wpq\") pod \"node-ca-7845q\" (UID: \"346c8e4c-4119-41a8-b798-c23d6de9b203\") " pod="openshift-image-registry/node-ca-7845q"
Sep 17 15:38:57.048508 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.048442    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzx86\" (UniqueName: \"kubernetes.io/projected/6588c63a-cfe8-415b-a31f-aecc29f7f69c-kube-api-access-dzx86\") pod \"multus-additional-cni-plugins-njl24\" (UID: \"6588c63a-cfe8-415b-a31f-aecc29f7f69c\") " pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.049259 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.049048    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm8gk\" (UniqueName: \"kubernetes.io/projected/cb06d52b-3ebb-4d34-a72e-c4a109808a4a-kube-api-access-vm8gk\") pod \"iptables-alerter-k8lsf\" (UID: \"cb06d52b-3ebb-4d34-a72e-c4a109808a4a\") " pod="openshift-network-operator/iptables-alerter-k8lsf"
Sep 17 15:38:57.049385 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.049235    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal" event={"ID":"6abc4d8c8ce3d693c97b0f59444f7f92","Type":"ContainerStarted","Data":"1b6eff5492280210b7f89f6e48021f39da5184f9037d9c524db383410a059ceb"}
Sep 17 15:38:57.051284 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.051255    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj79l\" (UniqueName: \"kubernetes.io/projected/34864426-aeff-4ea6-8bc1-145d6ee3f791-kube-api-access-nj79l\") pod \"aws-ebs-csi-driver-node-l8ld9\" (UID: \"34864426-aeff-4ea6-8bc1-145d6ee3f791\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.051529 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.051400    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal" event={"ID":"5723e815bf040e5d45b5f45f3fe46312","Type":"ContainerStarted","Data":"1eeec627f2d404cd3180cc75fc6241c18cf5a996190fbbebce93ddb708b99e3d"}
Sep 17 15:38:57.051943 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.051921    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ntrq\" (UniqueName: \"kubernetes.io/projected/3a29176b-44e8-445f-8ecb-9f6de6f9eb70-kube-api-access-7ntrq\") pod \"multus-bvgdz\" (UID: \"3a29176b-44e8-445f-8ecb-9f6de6f9eb70\") " pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.052372 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.052331    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92w8w\" (UniqueName: \"kubernetes.io/projected/cc3bbb2e-6060-4eca-89d0-33cc860b2039-kube-api-access-92w8w\") pod \"tuned-qr6th\" (UID: \"cc3bbb2e-6060-4eca-89d0-33cc860b2039\") " pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.053501 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.053426    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw6xk\" (UniqueName: \"kubernetes.io/projected/9f8f3b74-61b5-4b22-bf59-ab2ec9054df1-kube-api-access-kw6xk\") pod \"node-resolver-vzk9l\" (UID: \"9f8f3b74-61b5-4b22-bf59-ab2ec9054df1\") " pod="openshift-dns/node-resolver-vzk9l"
Sep 17 15:38:57.135964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.135915    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr\") pod \"network-check-target-twzfl\" (UID: \"dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7\") " pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:38:57.135964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.135967    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-slash\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136217 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.135996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvmc\" (UniqueName: \"kubernetes.io/projected/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-kube-api-access-5kvmc\") pod \"network-metrics-daemon-c4llb\" (UID: \"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf\") " pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:38:57.136217 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136058    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-slash\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136217 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136119    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-run-openvswitch\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136217 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136159    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-ovnkube-script-lib\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136217 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136188    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-run-openvswitch\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136217 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136194    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-run-ovn\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136232    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136270    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-run-ovn\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136269    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72f72846-2cc4-4793-a883-42382a787e30-dbus\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136305    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136335    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-ovn-node-metrics-cert\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136344    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136368    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-cni-bin\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136389    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-etc-openvswitch\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136413    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-env-overrides\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136440    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-systemd-units\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136465    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c35b1dbc-5400-46a6-a74a-1fac21cf6cff-agent-certs\") pod \"konnectivity-agent-9t4xp\" (UID: \"c35b1dbc-5400-46a6-a74a-1fac21cf6cff\") " pod="kube-system/konnectivity-agent-9t4xp"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136482    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-run-netns\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136515    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-log-socket\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136536    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-ovnkube-config\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.136550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136555    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7gs4\" (UniqueName: \"kubernetes.io/projected/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-kube-api-access-j7gs4\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136573    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-var-lib-openvswitch\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136588    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-cni-netd\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136611    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136648    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c35b1dbc-5400-46a6-a74a-1fac21cf6cff-konnectivity-ca\") pod \"konnectivity-agent-9t4xp\" (UID: \"c35b1dbc-5400-46a6-a74a-1fac21cf6cff\") " pod="kube-system/konnectivity-agent-9t4xp"
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-kubelet\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136705    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs\") pod \"network-metrics-daemon-c4llb\" (UID: \"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf\") " pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136728    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-node-log\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136762    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-run-systemd\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72f72846-2cc4-4793-a883-42382a787e30-kubelet-config\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136839    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72f72846-2cc4-4793-a883-42382a787e30-dbus\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136863    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-ovnkube-script-lib\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136900    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72f72846-2cc4-4793-a883-42382a787e30-kubelet-config\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.136941    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.137006    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.137029    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret podName:72f72846-2cc4-4793-a883-42382a787e30 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:57.63698935 +0000 UTC m=+3.108829180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret") pod "global-pull-secret-syncer-vt2l8" (UID: "72f72846-2cc4-4793-a883-42382a787e30") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.137061    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-node-log\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137296 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.137074    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs podName:e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf nodeName:}" failed. No retries permitted until 2026-09-17 15:38:57.637037573 +0000 UTC m=+3.108877405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs") pod "network-metrics-daemon-c4llb" (UID: "e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:57.137964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.136944    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-kubelet\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.137105    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-run-systemd\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.137148    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-systemd-units\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.137185    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-cni-bin\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.137223    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-etc-openvswitch\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.137307    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-run-netns\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.137321    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-ovnkube-config\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.137395    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c35b1dbc-5400-46a6-a74a-1fac21cf6cff-konnectivity-ca\") pod \"konnectivity-agent-9t4xp\" (UID: \"c35b1dbc-5400-46a6-a74a-1fac21cf6cff\") " pod="kube-system/konnectivity-agent-9t4xp"
Sep 17 15:38:57.137964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.137406    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-log-socket\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.137468    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-var-lib-openvswitch\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.137479    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.137510    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-host-cni-netd\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.137964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.137625    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-env-overrides\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.140237 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.140172    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c35b1dbc-5400-46a6-a74a-1fac21cf6cff-agent-certs\") pod \"konnectivity-agent-9t4xp\" (UID: \"c35b1dbc-5400-46a6-a74a-1fac21cf6cff\") " pod="kube-system/konnectivity-agent-9t4xp"
Sep 17 15:38:57.140863 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.140830    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-ovn-node-metrics-cert\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.142633 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.142449    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:57.142633 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.142485    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:57.142633 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.142500    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5b5fr for pod openshift-network-diagnostics/network-check-target-twzfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:57.142633 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.142591    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr podName:dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:57.642571548 +0000 UTC m=+3.114411384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5b5fr" (UniqueName: "kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr") pod "network-check-target-twzfl" (UID: "dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:57.146296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.146229    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kvmc\" (UniqueName: \"kubernetes.io/projected/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-kube-api-access-5kvmc\") pod \"network-metrics-daemon-c4llb\" (UID: \"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf\") " pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:38:57.147059 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.147034    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7gs4\" (UniqueName: \"kubernetes.io/projected/9afd46cf-843e-48bd-b3a2-7f24fceed8fc-kube-api-access-j7gs4\") pod \"ovnkube-node-hwfgx\" (UID: \"9afd46cf-843e-48bd-b3a2-7f24fceed8fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.222981 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.222903    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vzk9l"
Sep 17 15:38:57.232016 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.231983    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qr6th"
Sep 17 15:38:57.245171 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.245143    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7845q"
Sep 17 15:38:57.253104 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.253079    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9"
Sep 17 15:38:57.262043 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.262019    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k8lsf"
Sep 17 15:38:57.279995 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.279971    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-njl24"
Sep 17 15:38:57.290116 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.290087    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bvgdz"
Sep 17 15:38:57.298071 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.298050    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:38:57.303984 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.303964    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9t4xp"
Sep 17 15:38:57.310158 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.310137    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:57.642281 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.642191    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs\") pod \"network-metrics-daemon-c4llb\" (UID: \"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf\") " pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:38:57.642281 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.642257    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:38:57.642521 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.642330    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:57.642521 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.642362    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:57.642521 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.642413    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret podName:72f72846-2cc4-4793-a883-42382a787e30 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:58.642398082 +0000 UTC m=+4.114237914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret") pod "global-pull-secret-syncer-vt2l8" (UID: "72f72846-2cc4-4793-a883-42382a787e30") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:57.642521 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.642435    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs podName:e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf nodeName:}" failed. No retries permitted until 2026-09-17 15:38:58.642422453 +0000 UTC m=+4.114262281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs") pod "network-metrics-daemon-c4llb" (UID: "e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:57.742873 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.742833    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr\") pod \"network-check-target-twzfl\" (UID: \"dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7\") " pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:38:57.743062 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.743035    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:57.743141 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.743065    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:57.743141 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.743104    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5b5fr for pod openshift-network-diagnostics/network-check-target-twzfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:57.743243 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:57.743184    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr podName:dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:58.74316265 +0000 UTC m=+4.215002501 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5b5fr" (UniqueName: "kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr") pod "network-check-target-twzfl" (UID: "dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:57.970937 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.970830    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:33:57 +0000 UTC" deadline="2028-03-28 00:19:55.476598764 +0000 UTC"
Sep 17 15:38:57.970937 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:57.970892    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13376h40m57.505711432s"
Sep 17 15:38:58.193698 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:58.193655    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a29176b_44e8_445f_8ecb_9f6de6f9eb70.slice/crio-16c0d81e50945ce77d491f83acf9e66edd6b61307cae40ab3e6beed4d3db6601 WatchSource:0}: Error finding container 16c0d81e50945ce77d491f83acf9e66edd6b61307cae40ab3e6beed4d3db6601: Status 404 returned error can't find the container with id 16c0d81e50945ce77d491f83acf9e66edd6b61307cae40ab3e6beed4d3db6601
Sep 17 15:38:58.199232 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:58.198996    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3bbb2e_6060_4eca_89d0_33cc860b2039.slice/crio-d3c41cb7020c244683af5042b9d670c74326a8608e610a58fcc30dd949538a37 WatchSource:0}: Error finding container d3c41cb7020c244683af5042b9d670c74326a8608e610a58fcc30dd949538a37: Status 404 returned error can't find the container with id d3c41cb7020c244683af5042b9d670c74326a8608e610a58fcc30dd949538a37
Sep 17 15:38:58.222189 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:58.222157    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6588c63a_cfe8_415b_a31f_aecc29f7f69c.slice/crio-6526a3c2391b802431201987dc3edadb2bb739e1418d92405f6808a438d8cf6a WatchSource:0}: Error finding container 6526a3c2391b802431201987dc3edadb2bb739e1418d92405f6808a438d8cf6a: Status 404 returned error can't find the container with id 6526a3c2391b802431201987dc3edadb2bb739e1418d92405f6808a438d8cf6a
Sep 17 15:38:58.222811 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:58.222791    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34864426_aeff_4ea6_8bc1_145d6ee3f791.slice/crio-e387ab04e8d1d235573773c13f5530476e454126e63b5a892e5dc5c90aa34492 WatchSource:0}: Error finding container e387ab04e8d1d235573773c13f5530476e454126e63b5a892e5dc5c90aa34492: Status 404 returned error can't find the container with id e387ab04e8d1d235573773c13f5530476e454126e63b5a892e5dc5c90aa34492
Sep 17 15:38:58.223617 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:58.223590    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb06d52b_3ebb_4d34_a72e_c4a109808a4a.slice/crio-bcb4b47227da118d5811fd94d2bf80b0cc9e2cb62b2e7d8449340e2c9d9ba25e WatchSource:0}: Error finding container bcb4b47227da118d5811fd94d2bf80b0cc9e2cb62b2e7d8449340e2c9d9ba25e: Status 404 returned error can't find the container with id bcb4b47227da118d5811fd94d2bf80b0cc9e2cb62b2e7d8449340e2c9d9ba25e
Sep 17 15:38:58.224427 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:38:58.224354    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afd46cf_843e_48bd_b3a2_7f24fceed8fc.slice/crio-c21a4406a9461f329869d03c952828c063004020142aca9036c9a2a49e828829 WatchSource:0}: Error finding container c21a4406a9461f329869d03c952828c063004020142aca9036c9a2a49e828829: Status 404 returned error can't find the container with id c21a4406a9461f329869d03c952828c063004020142aca9036c9a2a49e828829
Sep 17 15:38:58.651046 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:58.650847    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:38:58.651046 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:58.650924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs\") pod \"network-metrics-daemon-c4llb\" (UID: \"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf\") " pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:38:58.651046 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:58.651049    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:58.651323 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:58.651059    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:58.651323 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:58.651113    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs podName:e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf nodeName:}" failed. No retries permitted until 2026-09-17 15:39:00.651094385 +0000 UTC m=+6.122934225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs") pod "network-metrics-daemon-c4llb" (UID: "e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:58.651323 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:58.651132    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret podName:72f72846-2cc4-4793-a883-42382a787e30 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:00.651122931 +0000 UTC m=+6.122962755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret") pod "global-pull-secret-syncer-vt2l8" (UID: "72f72846-2cc4-4793-a883-42382a787e30") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:58.752086 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:58.751998    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr\") pod \"network-check-target-twzfl\" (UID: \"dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7\") " pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:38:58.752258 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:58.752205    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:58.752258 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:58.752233    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:58.752258 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:58.752249    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5b5fr for pod openshift-network-diagnostics/network-check-target-twzfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:58.752432 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:58.752321    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr podName:dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:00.752301026 +0000 UTC m=+6.224140878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5b5fr" (UniqueName: "kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr") pod "network-check-target-twzfl" (UID: "dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:58.971150 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:58.971072    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:33:57 +0000 UTC" deadline="2028-05-03 04:36:23.139710375 +0000 UTC"
Sep 17 15:38:58.971150 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:58.971142    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14244h57m24.168573497s"
Sep 17 15:38:59.045218 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.043080    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:38:59.045218 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:59.043219    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:38:59.045218 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.043319    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:38:59.045218 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:59.043419    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:38:59.045218 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.043484    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:38:59.045218 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:38:59.043572    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:38:59.075252 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.075210    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qr6th" event={"ID":"cc3bbb2e-6060-4eca-89d0-33cc860b2039","Type":"ContainerStarted","Data":"d3c41cb7020c244683af5042b9d670c74326a8608e610a58fcc30dd949538a37"}
Sep 17 15:38:59.090198 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.090098    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bvgdz" event={"ID":"3a29176b-44e8-445f-8ecb-9f6de6f9eb70","Type":"ContainerStarted","Data":"16c0d81e50945ce77d491f83acf9e66edd6b61307cae40ab3e6beed4d3db6601"}
Sep 17 15:38:59.123945 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.123122    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal" event={"ID":"6abc4d8c8ce3d693c97b0f59444f7f92","Type":"ContainerStarted","Data":"5f80944a4ac7e184a2367913009c19aa40a2a7cb43a94de2d79d714162acfced"}
Sep 17 15:38:59.141002 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.139886    2574 generic.go:356] "Generic (PLEG): container finished" podID="5723e815bf040e5d45b5f45f3fe46312" containerID="54dee0911d640fe9c5e31de5d349984693b907be36e10892fb58e0006f0f590f" exitCode=0
Sep 17 15:38:59.141002 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.140605    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal" event={"ID":"5723e815bf040e5d45b5f45f3fe46312","Type":"ContainerDied","Data":"54dee0911d640fe9c5e31de5d349984693b907be36e10892fb58e0006f0f590f"}
Sep 17 15:38:59.141002 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.140947    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal" podStartSLOduration=3.140924768 podStartE2EDuration="3.140924768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:59.139240509 +0000 UTC m=+4.611080362" watchObservedRunningTime="2026-09-17 15:38:59.140924768 +0000 UTC m=+4.612764618"
Sep 17 15:38:59.161090 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.160952    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9t4xp" event={"ID":"c35b1dbc-5400-46a6-a74a-1fac21cf6cff","Type":"ContainerStarted","Data":"b945babe7a3301f44296a19d8ac36d38113fdd3037506889b0f041af7dcf0ae9"}
Sep 17 15:38:59.166359 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.166279    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7845q" event={"ID":"346c8e4c-4119-41a8-b798-c23d6de9b203","Type":"ContainerStarted","Data":"995e094d0a80da67443f82c19dc3174dd53df4cfc9595a8a4e201e3fb85c214a"}
Sep 17 15:38:59.191482 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.191444    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx" event={"ID":"9afd46cf-843e-48bd-b3a2-7f24fceed8fc","Type":"ContainerStarted","Data":"c21a4406a9461f329869d03c952828c063004020142aca9036c9a2a49e828829"}
Sep 17 15:38:59.218656 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.218572    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njl24" event={"ID":"6588c63a-cfe8-415b-a31f-aecc29f7f69c","Type":"ContainerStarted","Data":"6526a3c2391b802431201987dc3edadb2bb739e1418d92405f6808a438d8cf6a"}
Sep 17 15:38:59.227217 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.227139    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vzk9l" event={"ID":"9f8f3b74-61b5-4b22-bf59-ab2ec9054df1","Type":"ContainerStarted","Data":"f1c4bbaac2a1f2dccf1f045d2b50a336b067d8e8873865c03cd5f1803557f8c2"}
Sep 17 15:38:59.241493 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.241381    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k8lsf" event={"ID":"cb06d52b-3ebb-4d34-a72e-c4a109808a4a","Type":"ContainerStarted","Data":"bcb4b47227da118d5811fd94d2bf80b0cc9e2cb62b2e7d8449340e2c9d9ba25e"}
Sep 17 15:38:59.249076 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:38:59.249040    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9" event={"ID":"34864426-aeff-4ea6-8bc1-145d6ee3f791","Type":"ContainerStarted","Data":"e387ab04e8d1d235573773c13f5530476e454126e63b5a892e5dc5c90aa34492"}
Sep 17 15:39:00.273323 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:00.273202    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal" event={"ID":"5723e815bf040e5d45b5f45f3fe46312","Type":"ContainerStarted","Data":"4db34025ebb8f6d5abbba6829bc4184b20e77414d7c4d3ae6f1c1794165a004a"}
Sep 17 15:39:00.669889 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:00.669796    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:00.669889 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:00.669875    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs\") pod \"network-metrics-daemon-c4llb\" (UID: \"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf\") " pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:00.670105 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:00.670023    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:00.670105 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:00.670086    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs podName:e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf nodeName:}" failed. No retries permitted until 2026-09-17 15:39:04.670067391 +0000 UTC m=+10.141907223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs") pod "network-metrics-daemon-c4llb" (UID: "e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:00.670567 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:00.670512    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:00.670664 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:00.670589    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret podName:72f72846-2cc4-4793-a883-42382a787e30 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:04.670570741 +0000 UTC m=+10.142410573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret") pod "global-pull-secret-syncer-vt2l8" (UID: "72f72846-2cc4-4793-a883-42382a787e30") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:00.770831 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:00.770788    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr\") pod \"network-check-target-twzfl\" (UID: \"dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7\") " pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:00.771027 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:00.770977    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:00.771027 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:00.771025    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:00.771149 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:00.771037    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5b5fr for pod openshift-network-diagnostics/network-check-target-twzfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:00.771149 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:00.771100    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr podName:dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:04.771079705 +0000 UTC m=+10.242919531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5b5fr" (UniqueName: "kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr") pod "network-check-target-twzfl" (UID: "dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:01.053171 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:01.052642    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:01.053171 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:01.052767    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:01.053171 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:01.053098    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:01.053171 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:01.053111    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:01.053579 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:01.053199    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:01.053579 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:01.053287    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:03.048811 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:03.048076    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:03.048811 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:03.048090    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:03.048811 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:03.048212    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:03.048811 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:03.048313    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:03.048811 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:03.048356    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:03.048811 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:03.048432    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:04.707628 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:04.707589    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:04.708094 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:04.707659    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs\") pod \"network-metrics-daemon-c4llb\" (UID: \"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf\") " pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:04.708094 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:04.707782    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:04.708094 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:04.707861    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret podName:72f72846-2cc4-4793-a883-42382a787e30 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:12.707839845 +0000 UTC m=+18.179679680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret") pod "global-pull-secret-syncer-vt2l8" (UID: "72f72846-2cc4-4793-a883-42382a787e30") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:04.708094 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:04.707796    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:04.708094 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:04.707926    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs podName:e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf nodeName:}" failed. No retries permitted until 2026-09-17 15:39:12.707909023 +0000 UTC m=+18.179748855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs") pod "network-metrics-daemon-c4llb" (UID: "e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:04.808130 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:04.808090    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr\") pod \"network-check-target-twzfl\" (UID: \"dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7\") " pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:04.808301 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:04.808254    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:04.808301 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:04.808275    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:04.808301 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:04.808288    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5b5fr for pod openshift-network-diagnostics/network-check-target-twzfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:04.808487 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:04.808345    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr podName:dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:12.808326986 +0000 UTC m=+18.280166825 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5b5fr" (UniqueName: "kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr") pod "network-check-target-twzfl" (UID: "dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:05.047776 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:05.046240    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:05.047776 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:05.046379    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:05.047776 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:05.046820    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:05.047776 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:05.046957    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:05.047776 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:05.047013    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:05.047776 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:05.047113    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:07.043403 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:07.043271    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:07.043868 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:07.043401    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:07.043868 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:07.043278    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:07.043868 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:07.043782    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:07.044027 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:07.043880    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:07.044027 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:07.043953    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:09.042398 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:09.042368    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:09.042850 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:09.042375    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:09.042850 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:09.042482    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:09.042850 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:09.042495    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:09.042850 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:09.042570    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:09.042850 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:09.042646    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:11.042876 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:11.042643    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:11.043295 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:11.042643    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:11.043295 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:11.042986    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:11.043295 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:11.043024    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:11.043295 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:11.042674    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:11.043295 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:11.043101    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:12.766791 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:12.766744    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs\") pod \"network-metrics-daemon-c4llb\" (UID: \"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf\") " pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:12.767288 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:12.766836    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:12.767288 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:12.766917    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:12.767288 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:12.766943    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:12.767288 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:12.766982    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs podName:e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf nodeName:}" failed. No retries permitted until 2026-09-17 15:39:28.7669675 +0000 UTC m=+34.238807325 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs") pod "network-metrics-daemon-c4llb" (UID: "e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:12.767288 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:12.766996    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret podName:72f72846-2cc4-4793-a883-42382a787e30 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:28.766990457 +0000 UTC m=+34.238830280 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret") pod "global-pull-secret-syncer-vt2l8" (UID: "72f72846-2cc4-4793-a883-42382a787e30") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:12.867788 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:12.867756    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr\") pod \"network-check-target-twzfl\" (UID: \"dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7\") " pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:12.867977 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:12.867937    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:12.867977 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:12.867959    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:12.867977 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:12.867969    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5b5fr for pod openshift-network-diagnostics/network-check-target-twzfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:12.868135 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:12.868022    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr podName:dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:28.868006391 +0000 UTC m=+34.339846222 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5b5fr" (UniqueName: "kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr") pod "network-check-target-twzfl" (UID: "dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:13.045749 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:13.045640    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:13.045749 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:13.045663    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:13.045749 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:13.045640    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:13.045988 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:13.045755    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:13.045988 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:13.045827    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:13.045988 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:13.045927    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:15.043884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:15.043849    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:15.044338 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:15.043957    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:15.044338 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:15.044041    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:15.044338 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:15.044171    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:15.044338 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:15.044216    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:15.044338 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:15.044287    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:16.315705 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:16.315668    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qr6th" event={"ID":"cc3bbb2e-6060-4eca-89d0-33cc860b2039","Type":"ContainerStarted","Data":"1fe16adf418c47d419a9cbb5928f335ef3ec44a4dc2b1be8d5819692e87514f4"}
Sep 17 15:39:16.335948 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:16.335882    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal" podStartSLOduration=20.335861245 podStartE2EDuration="20.335861245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:00.2969826 +0000 UTC m=+5.768822449" watchObservedRunningTime="2026-09-17 15:39:16.335861245 +0000 UTC m=+21.807701093"
Sep 17 15:39:16.337595 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:16.337553    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qr6th" podStartSLOduration=3.377703122 podStartE2EDuration="21.337535235s" totalImagesPullingTime="17.959832113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:16.335014461 +0000 UTC m=+21.806854320" watchObservedRunningTime="2026-09-17 15:39:16.337535235 +0000 UTC m=+21.809375074"
Sep 17 15:39:17.042791 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.042549    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:17.043005 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.042553    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:17.043005 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:17.042902    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:17.043005 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.042557    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:17.043005 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:17.042974    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:17.043226 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:17.043057    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:17.318836 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.318791    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9" event={"ID":"34864426-aeff-4ea6-8bc1-145d6ee3f791","Type":"ContainerStarted","Data":"15d1a0695cfc39c839509d2e2a208b790dc64fa8f15ab3b1035755072e8b267e"}
Sep 17 15:39:17.322859 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.322814    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bvgdz" event={"ID":"3a29176b-44e8-445f-8ecb-9f6de6f9eb70","Type":"ContainerStarted","Data":"84118f233eeaf66122ea96faf9704796820a84621925719f9e0b5d3b81055408"}
Sep 17 15:39:17.324239 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.324206    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9t4xp" event={"ID":"c35b1dbc-5400-46a6-a74a-1fac21cf6cff","Type":"ContainerStarted","Data":"e30864af3c8387ee44a4ba2fda64a36a878f30dadcf61bf8cb8b734bd548b91a"}
Sep 17 15:39:17.325536 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.325510    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7845q" event={"ID":"346c8e4c-4119-41a8-b798-c23d6de9b203","Type":"ContainerStarted","Data":"c2e32f8866c4d0a7d0ea9e7d4afef6e121e4af40a39e07c76744129d696ce6b4"}
Sep 17 15:39:17.328575 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.328556    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx" event={"ID":"9afd46cf-843e-48bd-b3a2-7f24fceed8fc","Type":"ContainerStarted","Data":"014fad29064b591261a678e1819cf71a9ed2872166826d51ddcd6fd8c13ad732"}
Sep 17 15:39:17.328656 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.328615    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx" event={"ID":"9afd46cf-843e-48bd-b3a2-7f24fceed8fc","Type":"ContainerStarted","Data":"1ddd894e3babedc40358d4b3ddf3efd45cbdf831a8c3f1ddb08176ddea227336"}
Sep 17 15:39:17.328656 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.328647    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx" event={"ID":"9afd46cf-843e-48bd-b3a2-7f24fceed8fc","Type":"ContainerStarted","Data":"28f302fa09d731a94268ba57063ab9f5831771cf0929144856b985f88c41bb85"}
Sep 17 15:39:17.328732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.328682    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx" event={"ID":"9afd46cf-843e-48bd-b3a2-7f24fceed8fc","Type":"ContainerStarted","Data":"5c4122b73b0d865533eac8adb8ba2c4d866b47d456fa21280d6a078b03abc947"}
Sep 17 15:39:17.328732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.328715    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx" event={"ID":"9afd46cf-843e-48bd-b3a2-7f24fceed8fc","Type":"ContainerStarted","Data":"1338a6f0de8c15107d386ceba480fb126b6e9066f905f7d90e56a8ca0e482276"}
Sep 17 15:39:17.328807 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.328751    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx" event={"ID":"9afd46cf-843e-48bd-b3a2-7f24fceed8fc","Type":"ContainerStarted","Data":"6f7eb69fd4fb7e52aa549528173dd343c85b7995f7b89137ea255076a16d86ab"}
Sep 17 15:39:17.330136 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.330116    2574 generic.go:356] "Generic (PLEG): container finished" podID="6588c63a-cfe8-415b-a31f-aecc29f7f69c" containerID="d72a4ed8511a8885cdc85182ee83387b3c52a93278c9ddd1b2383086497bf257" exitCode=0
Sep 17 15:39:17.330195 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.330178    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njl24" event={"ID":"6588c63a-cfe8-415b-a31f-aecc29f7f69c","Type":"ContainerDied","Data":"d72a4ed8511a8885cdc85182ee83387b3c52a93278c9ddd1b2383086497bf257"}
Sep 17 15:39:17.331571 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.331551    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vzk9l" event={"ID":"9f8f3b74-61b5-4b22-bf59-ab2ec9054df1","Type":"ContainerStarted","Data":"5878f98947a3b80e1e4fba6d90d8399e234f80bf36be00ebb8fe69dbb7aa2055"}
Sep 17 15:39:17.342178 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.342141    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bvgdz" podStartSLOduration=3.30008963 podStartE2EDuration="21.342129726s" totalImagesPullingTime="18.042040096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:17.341991297 +0000 UTC m=+22.813831143" watchObservedRunningTime="2026-09-17 15:39:17.342129726 +0000 UTC m=+22.813969569"
Sep 17 15:39:17.370093 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.370055    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vzk9l" podStartSLOduration=4.43822044 podStartE2EDuration="22.370041965s" totalImagesPullingTime="17.931821525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:17.369978464 +0000 UTC m=+22.841818314" watchObservedRunningTime="2026-09-17 15:39:17.370041965 +0000 UTC m=+22.841881803"
Sep 17 15:39:17.370612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.370571    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7845q" podStartSLOduration=4.438222873 podStartE2EDuration="22.370562998s" totalImagesPullingTime="17.932340125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:17.355884355 +0000 UTC m=+22.827724200" watchObservedRunningTime="2026-09-17 15:39:17.370562998 +0000 UTC m=+22.842402845"
Sep 17 15:39:17.411038 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.411003    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9t4xp" podStartSLOduration=10.57601999 podStartE2EDuration="21.410991229s" totalImagesPullingTime="10.834971239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:17.410636545 +0000 UTC m=+22.882476395" watchObservedRunningTime="2026-09-17 15:39:17.410991229 +0000 UTC m=+22.882831075"
Sep 17 15:39:17.484815 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.484736    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:17.544128 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.544109    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 15:39:17.563089 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:17.563066    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:18.012375 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:18.012326    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:18.081318 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:18.081290    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:18.126955 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:18.126848    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T15:39:17.544125914Z","UUID":"65f84c2e-c867-4292-9a8d-3fe1e70d4bdd","Handler":null,"Name":"","Endpoint":""}
Sep 17 15:39:18.128853 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:18.128831    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 15:39:18.128853 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:18.128868    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 15:39:18.335744 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:18.335629    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k8lsf" event={"ID":"cb06d52b-3ebb-4d34-a72e-c4a109808a4a","Type":"ContainerStarted","Data":"78aaca3965b5c2d61cc9f0c95733cfa27e07922a36ffeedd75af431bd9527be7"}
Sep 17 15:39:18.338022 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:18.337996    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9" event={"ID":"34864426-aeff-4ea6-8bc1-145d6ee3f791","Type":"ContainerStarted","Data":"3939020b7c8b9dab79378a0d5e6960a06f25b34839fe8a3e8af8048fbfd9e382"}
Sep 17 15:39:18.350909 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:18.350855    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k8lsf" podStartSLOduration=5.414055914 podStartE2EDuration="23.350835768s" totalImagesPullingTime="17.936779854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:18.350283924 +0000 UTC m=+23.822123771" watchObservedRunningTime="2026-09-17 15:39:18.350835768 +0000 UTC m=+23.822675615"
Sep 17 15:39:18.470321 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:18.470288    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:18.816506 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:18.816284    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:18.893660 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:18.893629    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:19.042538 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:19.042413    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:19.042538 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:19.042465    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:19.042749 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:19.042559    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:19.042749 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:19.042566    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:19.042846 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:19.042746    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:19.042846 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:19.042828    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:19.342882 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:19.342804    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9" event={"ID":"34864426-aeff-4ea6-8bc1-145d6ee3f791","Type":"ContainerStarted","Data":"475dee96ba274f129de68263db5614fd74b71cbc39004c513f58d8cb2d29d5fe"}
Sep 17 15:39:19.347123 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:19.347095    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx" event={"ID":"9afd46cf-843e-48bd-b3a2-7f24fceed8fc","Type":"ContainerStarted","Data":"f49d721367db483971bfbd9c9e67acd556b98ca517c9ee91834a6fc49b61695a"}
Sep 17 15:39:19.362089 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:19.362029    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8ld9" podStartSLOduration=4.257588407 podStartE2EDuration="24.362009851s" totalImagesPullingTime="20.104421444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:19.361513169 +0000 UTC m=+24.833353015" watchObservedRunningTime="2026-09-17 15:39:19.362009851 +0000 UTC m=+24.833849699"
Sep 17 15:39:20.730954 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:20.730905    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:20.831798 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:20.831768    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:21.042529 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:21.042452    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:21.042682 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:21.042451    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:21.042682 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:21.042572    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:21.042682 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:21.042601    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:21.042839 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:21.042712    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:21.042839 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:21.042784    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:21.400605 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:21.400570    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9t4xp"
Sep 17 15:39:21.401265 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:21.401240    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9t4xp"
Sep 17 15:39:22.356960 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:22.356772    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx" event={"ID":"9afd46cf-843e-48bd-b3a2-7f24fceed8fc","Type":"ContainerStarted","Data":"8507a599d05de1f6fbdbb56283dd5fe5c764d45aaaf915f785edd8fe0d7de71a"}
Sep 17 15:39:22.357454 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:22.357069    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:39:22.357454 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:22.357139    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:39:22.357454 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:22.357181    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:39:22.358915 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:22.358892    2574 generic.go:356] "Generic (PLEG): container finished" podID="6588c63a-cfe8-415b-a31f-aecc29f7f69c" containerID="4366e542b7d7c7e64fb04b1440fe6b1f2d5b16a8d8d89e241e5992c7effb323d" exitCode=0
Sep 17 15:39:22.359027 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:22.358930    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njl24" event={"ID":"6588c63a-cfe8-415b-a31f-aecc29f7f69c","Type":"ContainerDied","Data":"4366e542b7d7c7e64fb04b1440fe6b1f2d5b16a8d8d89e241e5992c7effb323d"}
Sep 17 15:39:22.372776 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:22.372753    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:39:22.372876 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:22.372848    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:39:22.399531 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:22.399482    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx" podStartSLOduration=8.406123536 podStartE2EDuration="26.399465868s" totalImagesPullingTime="17.993342332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:22.399014518 +0000 UTC m=+27.870854376" watchObservedRunningTime="2026-09-17 15:39:22.399465868 +0000 UTC m=+27.871305715"
Sep 17 15:39:23.042720 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:23.042686    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:23.042720 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:23.042707    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:23.042874 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:23.042791    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:23.042922 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:23.042868    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:23.042922 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:23.042887    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:23.043014 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:23.042987    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:23.363398 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:23.363367    2574 generic.go:356] "Generic (PLEG): container finished" podID="6588c63a-cfe8-415b-a31f-aecc29f7f69c" containerID="9b5bb9d1318f4ebe909f128deb04b091d4b5ad0f1412d8f259c313752e683af7" exitCode=0
Sep 17 15:39:23.364172 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:23.363523    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njl24" event={"ID":"6588c63a-cfe8-415b-a31f-aecc29f7f69c","Type":"ContainerDied","Data":"9b5bb9d1318f4ebe909f128deb04b091d4b5ad0f1412d8f259c313752e683af7"}
Sep 17 15:39:23.610425 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:23.610386    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vt2l8"]
Sep 17 15:39:23.610612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:23.610592    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:23.610714 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:23.610692    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:23.613298 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:23.613273    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c4llb"]
Sep 17 15:39:23.613432 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:23.613389    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:23.613545 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:23.613519    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:23.614173 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:23.614153    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-twzfl"]
Sep 17 15:39:23.614247 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:23.614234    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:23.614327 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:23.614305    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:24.368631 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:24.368407    2574 generic.go:356] "Generic (PLEG): container finished" podID="6588c63a-cfe8-415b-a31f-aecc29f7f69c" containerID="76c07fcedecd797795f5a52a235693d047c1323745b75bc7cd8ae78497d83921" exitCode=0
Sep 17 15:39:24.369192 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:24.368473    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njl24" event={"ID":"6588c63a-cfe8-415b-a31f-aecc29f7f69c","Type":"ContainerDied","Data":"76c07fcedecd797795f5a52a235693d047c1323745b75bc7cd8ae78497d83921"}
Sep 17 15:39:25.043494 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:25.043462    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:25.043494 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:25.043482    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:25.043633 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:25.043571    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:25.043633 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:25.043613    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:25.043633 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:25.043624    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:25.043740 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:25.043705    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:27.034675 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:27.034634    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9t4xp"
Sep 17 15:39:27.035129 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:27.034823    2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:39:27.035365 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:27.035320    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9t4xp"
Sep 17 15:39:27.042780 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:27.042707    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:27.042901 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:27.042706    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:27.042901 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:27.042832    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:27.042901 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:27.042871    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:27.043044 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:27.042965    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:27.043113 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:27.043041    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:27.098456 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:27.098404    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:28.785390 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:28.785357    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs\") pod \"network-metrics-daemon-c4llb\" (UID: \"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf\") " pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:28.785818 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:28.785409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:28.785818 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:28.785509    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:28.785818 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:28.785511    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:28.785818 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:28.785557    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret podName:72f72846-2cc4-4793-a883-42382a787e30 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:00.785543872 +0000 UTC m=+66.257383697 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret") pod "global-pull-secret-syncer-vt2l8" (UID: "72f72846-2cc4-4793-a883-42382a787e30") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:28.785818 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:28.785571    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs podName:e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf nodeName:}" failed. No retries permitted until 2026-09-17 15:40:00.785564011 +0000 UTC m=+66.257403835 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs") pod "network-metrics-daemon-c4llb" (UID: "e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:28.886433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:28.886403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr\") pod \"network-check-target-twzfl\" (UID: \"dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7\") " pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:28.886595 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:28.886556    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:28.886595 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:28.886577    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:28.886595 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:28.886589    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5b5fr for pod openshift-network-diagnostics/network-check-target-twzfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:28.886689 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:28.886635    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr podName:dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:00.886621547 +0000 UTC m=+66.358461371 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5b5fr" (UniqueName: "kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr") pod "network-check-target-twzfl" (UID: "dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:29.042631 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:29.042580    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:29.042631 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:29.042597    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:29.042810 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:29.042583    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:29.042810 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:29.042691    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c4llb" podUID="e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf"
Sep 17 15:39:29.042810 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:29.042772    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vt2l8" podUID="72f72846-2cc4-4793-a883-42382a787e30"
Sep 17 15:39:29.042921 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:29.042835    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twzfl" podUID="dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7"
Sep 17 15:39:29.385583 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:29.385546    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njl24" event={"ID":"6588c63a-cfe8-415b-a31f-aecc29f7f69c","Type":"ContainerStarted","Data":"7474a6bd287406ecdba8a1e14994254caa27fdbfc2fbb581e00a9a26e4e8853a"}
Sep 17 15:39:30.270652 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.270626    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeReady"
Sep 17 15:39:30.271016 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.270740    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 15:39:30.342784 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.342752    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lpgls"]
Sep 17 15:39:30.357997 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.357974    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d7xgf"]
Sep 17 15:39:30.358104 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.358088    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lpgls"
Sep 17 15:39:30.361106 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.361075    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 15:39:30.361220 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.361126    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 15:39:30.361428 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.361413    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lggbn\""
Sep 17 15:39:30.361775 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.361763    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 15:39:30.371932 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.371909    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lpgls"]
Sep 17 15:39:30.372012 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.372001    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:30.374034 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.374012    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d7xgf"]
Sep 17 15:39:30.383293 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.383275    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 15:39:30.385359 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.385344    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 15:39:30.386470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.386454    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9xnvp\""
Sep 17 15:39:30.390660 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.390639    2574 generic.go:356] "Generic (PLEG): container finished" podID="6588c63a-cfe8-415b-a31f-aecc29f7f69c" containerID="7474a6bd287406ecdba8a1e14994254caa27fdbfc2fbb581e00a9a26e4e8853a" exitCode=0
Sep 17 15:39:30.390740 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.390679    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njl24" event={"ID":"6588c63a-cfe8-415b-a31f-aecc29f7f69c","Type":"ContainerDied","Data":"7474a6bd287406ecdba8a1e14994254caa27fdbfc2fbb581e00a9a26e4e8853a"}
Sep 17 15:39:30.495674 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.495656    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-config-volume\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:30.495769 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.495680    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:30.495769 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.495700    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9zl\" (UniqueName: \"kubernetes.io/projected/02f7fe3e-05c4-49ac-97d2-35337336c9cb-kube-api-access-dw9zl\") pod \"ingress-canary-lpgls\" (UID: \"02f7fe3e-05c4-49ac-97d2-35337336c9cb\") " pod="openshift-ingress-canary/ingress-canary-lpgls"
Sep 17 15:39:30.495910 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.495890    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-tmp-dir\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:30.495949 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.495922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2rc2\" (UniqueName: \"kubernetes.io/projected/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-kube-api-access-b2rc2\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:30.495949 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.495939    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02f7fe3e-05c4-49ac-97d2-35337336c9cb-cert\") pod \"ingress-canary-lpgls\" (UID: \"02f7fe3e-05c4-49ac-97d2-35337336c9cb\") " pod="openshift-ingress-canary/ingress-canary-lpgls"
Sep 17 15:39:30.574976 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.574951    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:30.596824 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.596801    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-tmp-dir\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:30.596824 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.596828    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2rc2\" (UniqueName: \"kubernetes.io/projected/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-kube-api-access-b2rc2\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:30.596972 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.596845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02f7fe3e-05c4-49ac-97d2-35337336c9cb-cert\") pod \"ingress-canary-lpgls\" (UID: \"02f7fe3e-05c4-49ac-97d2-35337336c9cb\") " pod="openshift-ingress-canary/ingress-canary-lpgls"
Sep 17 15:39:30.597039 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.597015    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-config-volume\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:30.597073 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.597054    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:30.597117 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.597081    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9zl\" (UniqueName: \"kubernetes.io/projected/02f7fe3e-05c4-49ac-97d2-35337336c9cb-kube-api-access-dw9zl\") pod \"ingress-canary-lpgls\" (UID: \"02f7fe3e-05c4-49ac-97d2-35337336c9cb\") " pod="openshift-ingress-canary/ingress-canary-lpgls"
Sep 17 15:39:30.597169 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.597118    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-tmp-dir\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:30.597358 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:30.597187    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:30.597425 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:30.597414    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls podName:300fa6c7-126b-4872-bb0f-13ca5a4eca5c nodeName:}" failed. No retries permitted until 2026-09-17 15:39:31.097392957 +0000 UTC m=+36.569232786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls") pod "dns-default-d7xgf" (UID: "300fa6c7-126b-4872-bb0f-13ca5a4eca5c") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:30.597678 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.597661    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-config-volume\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:30.600872 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.600857    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02f7fe3e-05c4-49ac-97d2-35337336c9cb-cert\") pod \"ingress-canary-lpgls\" (UID: \"02f7fe3e-05c4-49ac-97d2-35337336c9cb\") " pod="openshift-ingress-canary/ingress-canary-lpgls"
Sep 17 15:39:30.617538 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.617521    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2rc2\" (UniqueName: \"kubernetes.io/projected/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-kube-api-access-b2rc2\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:30.620038 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.620022    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9zl\" (UniqueName: \"kubernetes.io/projected/02f7fe3e-05c4-49ac-97d2-35337336c9cb-kube-api-access-dw9zl\") pod \"ingress-canary-lpgls\" (UID: \"02f7fe3e-05c4-49ac-97d2-35337336c9cb\") " pod="openshift-ingress-canary/ingress-canary-lpgls"
Sep 17 15:39:30.662282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.662259    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:30.667874 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.667857    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lpgls"
Sep 17 15:39:30.956487 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:30.956461    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lpgls"]
Sep 17 15:39:30.959969 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:39:30.959937    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f7fe3e_05c4_49ac_97d2_35337336c9cb.slice/crio-92cceac3379303d44f17636d55e5a47d0d060ef031c2e02323f5d6eaa0eec4df WatchSource:0}: Error finding container 92cceac3379303d44f17636d55e5a47d0d060ef031c2e02323f5d6eaa0eec4df: Status 404 returned error can't find the container with id 92cceac3379303d44f17636d55e5a47d0d060ef031c2e02323f5d6eaa0eec4df
Sep 17 15:39:31.042906 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:31.042877    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:39:31.042906 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:31.042894    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:39:31.043081 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:31.042883    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:39:31.049025 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:31.048990    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:39:31.049025 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:31.049013    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:39:31.049198 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:31.048999    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:39:31.049198 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:31.049129    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xkd6d\""
Sep 17 15:39:31.049198 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:31.049130    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:39:31.049329 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:31.049298    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j5497\""
Sep 17 15:39:31.101652 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:31.101626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:31.101765 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:31.101752    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:31.101813 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:31.101805    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls podName:300fa6c7-126b-4872-bb0f-13ca5a4eca5c nodeName:}" failed. No retries permitted until 2026-09-17 15:39:32.101789346 +0000 UTC m=+37.573629175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls") pod "dns-default-d7xgf" (UID: "300fa6c7-126b-4872-bb0f-13ca5a4eca5c") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:31.401665 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:31.401628    2574 generic.go:356] "Generic (PLEG): container finished" podID="6588c63a-cfe8-415b-a31f-aecc29f7f69c" containerID="487f7becb8d491c2a18640be8cf6817c773b2c747f9bf294c1a0d38576f2be6f" exitCode=0
Sep 17 15:39:31.402429 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:31.401705    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njl24" event={"ID":"6588c63a-cfe8-415b-a31f-aecc29f7f69c","Type":"ContainerDied","Data":"487f7becb8d491c2a18640be8cf6817c773b2c747f9bf294c1a0d38576f2be6f"}
Sep 17 15:39:31.403810 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:31.403781    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lpgls" event={"ID":"02f7fe3e-05c4-49ac-97d2-35337336c9cb","Type":"ContainerStarted","Data":"92cceac3379303d44f17636d55e5a47d0d060ef031c2e02323f5d6eaa0eec4df"}
Sep 17 15:39:32.108155 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:32.107910    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:32.108329 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:32.108074    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:32.108329 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:32.108261    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls podName:300fa6c7-126b-4872-bb0f-13ca5a4eca5c nodeName:}" failed. No retries permitted until 2026-09-17 15:39:34.108237034 +0000 UTC m=+39.580076875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls") pod "dns-default-d7xgf" (UID: "300fa6c7-126b-4872-bb0f-13ca5a4eca5c") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:32.409694 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:32.409625    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njl24" event={"ID":"6588c63a-cfe8-415b-a31f-aecc29f7f69c","Type":"ContainerStarted","Data":"08bfd4ebd812fcdff6fd2e5aed6f55592d05c9909a5dfff358e08835bb674520"}
Sep 17 15:39:32.597596 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:32.597558    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:32.680291 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:32.680160    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:34.124736 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:34.124704    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:34.125103 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:34.124828    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:34.125103 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:34.124891    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls podName:300fa6c7-126b-4872-bb0f-13ca5a4eca5c nodeName:}" failed. No retries permitted until 2026-09-17 15:39:38.124877288 +0000 UTC m=+43.596717116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls") pod "dns-default-d7xgf" (UID: "300fa6c7-126b-4872-bb0f-13ca5a4eca5c") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:34.176345 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:34.176316    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:34.239046 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:34.239014    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:34.415056 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:34.414978    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lpgls" event={"ID":"02f7fe3e-05c4-49ac-97d2-35337336c9cb","Type":"ContainerStarted","Data":"9a645418811e3cb28fd2539cba8f4a8a4b47842d072cc2c595ddc528dc644979"}
Sep 17 15:39:34.433676 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:34.433631    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lpgls" podStartSLOduration=1.7353683819999999 podStartE2EDuration="4.433615321s" totalImagesPullingTime="2.698246939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:34.433329746 +0000 UTC m=+39.905169593" watchObservedRunningTime="2026-09-17 15:39:34.433615321 +0000 UTC m=+39.905455161"
Sep 17 15:39:34.434072 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:34.434046    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-njl24" podStartSLOduration=11.074877045 podStartE2EDuration="39.434039344s" totalImagesPullingTime="28.359162299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:32.442655566 +0000 UTC m=+37.914495412" watchObservedRunningTime="2026-09-17 15:39:34.434039344 +0000 UTC m=+39.905879187"
Sep 17 15:39:38.151819 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:38.151778    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:38.152295 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:38.151959    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:38.152295 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:38.152041    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls podName:300fa6c7-126b-4872-bb0f-13ca5a4eca5c nodeName:}" failed. No retries permitted until 2026-09-17 15:39:46.152020247 +0000 UTC m=+51.623860076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls") pod "dns-default-d7xgf" (UID: "300fa6c7-126b-4872-bb0f-13ca5a4eca5c") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:38.254403 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:38.254368    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:38.315799 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:38.315769    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:38.371235 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:38.371203    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:42.009275 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:42.009236    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:42.076096 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:42.076057    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:46.209417 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:46.209372    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:39:46.209897 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:46.209536    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:46.209897 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:46.209600    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls podName:300fa6c7-126b-4872-bb0f-13ca5a4eca5c nodeName:}" failed. No retries permitted until 2026-09-17 15:40:02.209585123 +0000 UTC m=+67.681424951 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls") pod "dns-default-d7xgf" (UID: "300fa6c7-126b-4872-bb0f-13ca5a4eca5c") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:52.201009 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.200973    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-64b88459dd-xdnj5"]
Sep 17 15:39:52.202770 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.202755    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.207271 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.207249    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 15:39:52.207771 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.207744    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 15:39:52.208144 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.208127    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 15:39:52.208144 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.208151    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 15:39:52.208144 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.208160    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 15:39:52.208144 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.208159    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 15:39:52.211285 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.211253    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zswsd\""
Sep 17 15:39:52.224725 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.224698    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64b88459dd-xdnj5"]
Sep 17 15:39:52.350736 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.350697    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.350736 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.350734    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.350986 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.350878    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-default-certificate\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.350986 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.350915    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-stats-auth\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.350986 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.350939    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xz88\" (UniqueName: \"kubernetes.io/projected/1d1c6522-62ea-4c17-b992-33f236073899-kube-api-access-2xz88\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.422456 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.422408    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:52.451486 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.451374    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-default-certificate\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.451486 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.451407    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-stats-auth\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.451486 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.451426    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xz88\" (UniqueName: \"kubernetes.io/projected/1d1c6522-62ea-4c17-b992-33f236073899-kube-api-access-2xz88\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.451486 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.451466    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.451783 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:52.451584    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle podName:1d1c6522-62ea-4c17-b992-33f236073899 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:52.951568986 +0000 UTC m=+58.423408810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle") pod "router-default-64b88459dd-xdnj5" (UID: "1d1c6522-62ea-4c17-b992-33f236073899") : configmap references non-existent config key: service-ca.crt
Sep 17 15:39:52.451783 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.451667    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.451863 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:52.451831    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:39:52.451892 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:52.451879    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs podName:1d1c6522-62ea-4c17-b992-33f236073899 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:52.951863623 +0000 UTC m=+58.423703448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs") pod "router-default-64b88459dd-xdnj5" (UID: "1d1c6522-62ea-4c17-b992-33f236073899") : secret "router-metrics-certs-default" not found
Sep 17 15:39:52.454767 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.454714    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-default-certificate\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.454767 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.454731    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-stats-auth\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.459884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.459863    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xz88\" (UniqueName: \"kubernetes.io/projected/1d1c6522-62ea-4c17-b992-33f236073899-kube-api-access-2xz88\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.521231 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.521183    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:52.955783 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.955746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.955991 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:52.955791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:52.955991 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:52.955932    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle podName:1d1c6522-62ea-4c17-b992-33f236073899 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:53.95591406 +0000 UTC m=+59.427753907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle") pod "router-default-64b88459dd-xdnj5" (UID: "1d1c6522-62ea-4c17-b992-33f236073899") : configmap references non-existent config key: service-ca.crt
Sep 17 15:39:52.955991 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:52.955944    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:39:52.955991 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:52.955984    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs podName:1d1c6522-62ea-4c17-b992-33f236073899 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:53.955974286 +0000 UTC m=+59.427814110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs") pod "router-default-64b88459dd-xdnj5" (UID: "1d1c6522-62ea-4c17-b992-33f236073899") : secret "router-metrics-certs-default" not found
Sep 17 15:39:53.963918 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:53.963884    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:53.963918 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:53.963921    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:53.964371 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:53.964062    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle podName:1d1c6522-62ea-4c17-b992-33f236073899 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:55.964045512 +0000 UTC m=+61.435885338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle") pod "router-default-64b88459dd-xdnj5" (UID: "1d1c6522-62ea-4c17-b992-33f236073899") : configmap references non-existent config key: service-ca.crt
Sep 17 15:39:53.964371 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:53.964068    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:39:53.964371 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:53.964103    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs podName:1d1c6522-62ea-4c17-b992-33f236073899 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:55.964096497 +0000 UTC m=+61.435936320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs") pod "router-default-64b88459dd-xdnj5" (UID: "1d1c6522-62ea-4c17-b992-33f236073899") : secret "router-metrics-certs-default" not found
Sep 17 15:39:54.385072 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:54.385045    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hwfgx"
Sep 17 15:39:54.510916 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:54.510884    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:54.604415 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:54.604381    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:54.706879 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:54.706789    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:55.057936 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:55.057852    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:55.162429 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:55.162391    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:55.980262 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:55.980199    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:55.980262 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:55.980250    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:39:55.980517 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:55.980364    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:39:55.980517 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:55.980391    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle podName:1d1c6522-62ea-4c17-b992-33f236073899 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:59.980372373 +0000 UTC m=+65.452212221 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle") pod "router-default-64b88459dd-xdnj5" (UID: "1d1c6522-62ea-4c17-b992-33f236073899") : configmap references non-existent config key: service-ca.crt
Sep 17 15:39:55.980517 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:55.980416    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs podName:1d1c6522-62ea-4c17-b992-33f236073899 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:59.980409588 +0000 UTC m=+65.452249412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs") pod "router-default-64b88459dd-xdnj5" (UID: "1d1c6522-62ea-4c17-b992-33f236073899") : secret "router-metrics-certs-default" not found
Sep 17 15:39:56.072514 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.072477    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk"]
Sep 17 15:39:56.075119 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.075096    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk"
Sep 17 15:39:56.076977 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.076954    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-tsclf\""
Sep 17 15:39:56.077255 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.077237    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 15:39:56.077526 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.077511    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 15:39:56.077605 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.077526    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 15:39:56.077605 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.077536    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:39:56.085117 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.085099    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk"]
Sep 17 15:39:56.181488 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.181435    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57985d52-f04f-4487-84de-ddfd56e490af-serving-cert\") pod \"service-ca-operator-cf7c776b7-mf6mk\" (UID: \"57985d52-f04f-4487-84de-ddfd56e490af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk"
Sep 17 15:39:56.181488 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.181485    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57985d52-f04f-4487-84de-ddfd56e490af-config\") pod \"service-ca-operator-cf7c776b7-mf6mk\" (UID: \"57985d52-f04f-4487-84de-ddfd56e490af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk"
Sep 17 15:39:56.181735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.181588    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncw44\" (UniqueName: \"kubernetes.io/projected/57985d52-f04f-4487-84de-ddfd56e490af-kube-api-access-ncw44\") pod \"service-ca-operator-cf7c776b7-mf6mk\" (UID: \"57985d52-f04f-4487-84de-ddfd56e490af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk"
Sep 17 15:39:56.282650 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.282557    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57985d52-f04f-4487-84de-ddfd56e490af-serving-cert\") pod \"service-ca-operator-cf7c776b7-mf6mk\" (UID: \"57985d52-f04f-4487-84de-ddfd56e490af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk"
Sep 17 15:39:56.282650 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.282598    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57985d52-f04f-4487-84de-ddfd56e490af-config\") pod \"service-ca-operator-cf7c776b7-mf6mk\" (UID: \"57985d52-f04f-4487-84de-ddfd56e490af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk"
Sep 17 15:39:56.282860 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.282794    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncw44\" (UniqueName: \"kubernetes.io/projected/57985d52-f04f-4487-84de-ddfd56e490af-kube-api-access-ncw44\") pod \"service-ca-operator-cf7c776b7-mf6mk\" (UID: \"57985d52-f04f-4487-84de-ddfd56e490af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk"
Sep 17 15:39:56.283154 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.283130    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57985d52-f04f-4487-84de-ddfd56e490af-config\") pod \"service-ca-operator-cf7c776b7-mf6mk\" (UID: \"57985d52-f04f-4487-84de-ddfd56e490af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk"
Sep 17 15:39:56.284796 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.284778    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57985d52-f04f-4487-84de-ddfd56e490af-serving-cert\") pod \"service-ca-operator-cf7c776b7-mf6mk\" (UID: \"57985d52-f04f-4487-84de-ddfd56e490af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk"
Sep 17 15:39:56.291541 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.291523    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncw44\" (UniqueName: \"kubernetes.io/projected/57985d52-f04f-4487-84de-ddfd56e490af-kube-api-access-ncw44\") pod \"service-ca-operator-cf7c776b7-mf6mk\" (UID: \"57985d52-f04f-4487-84de-ddfd56e490af\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk"
Sep 17 15:39:56.383641 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.383600    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk"
Sep 17 15:39:56.650148 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:56.650116    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk"]
Sep 17 15:39:56.653485 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:39:56.653431    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57985d52_f04f_4487_84de_ddfd56e490af.slice/crio-096c0c1385fb4da93fb347bd6a2e18afd858148fc344b1da5f2990a0b657bc6f WatchSource:0}: Error finding container 096c0c1385fb4da93fb347bd6a2e18afd858148fc344b1da5f2990a0b657bc6f: Status 404 returned error can't find the container with id 096c0c1385fb4da93fb347bd6a2e18afd858148fc344b1da5f2990a0b657bc6f
Sep 17 15:39:57.470200 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:57.470163    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk" event={"ID":"57985d52-f04f-4487-84de-ddfd56e490af","Type":"ContainerStarted","Data":"096c0c1385fb4da93fb347bd6a2e18afd858148fc344b1da5f2990a0b657bc6f"}
Sep 17 15:39:58.014824 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.014792    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:58.107150 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.107116    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:58.141676 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.141648    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7sgh7"]
Sep 17 15:39:58.143632 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.143613    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.146080 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.146060    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 15:39:58.146202 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.146058    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 15:39:58.146883 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.146869    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8w8t9\""
Sep 17 15:39:58.146945 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.146916    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 15:39:58.146989 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.146941    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 15:39:58.155061 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.155037    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7sgh7"]
Sep 17 15:39:58.194503 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.194435    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:58.298874 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.298795    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59ps\" (UniqueName: \"kubernetes.io/projected/117720a7-6661-4079-81b4-f5b65382cfb6-kube-api-access-r59ps\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.298874 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.298830    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/117720a7-6661-4079-81b4-f5b65382cfb6-data-volume\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.298874 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.298853    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/117720a7-6661-4079-81b4-f5b65382cfb6-crio-socket\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.299078 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.298905    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.299078 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.299003    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.400033 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.399992    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.400211 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.400084    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.400211 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:58.400135    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:39:58.400312 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.400143    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59ps\" (UniqueName: \"kubernetes.io/projected/117720a7-6661-4079-81b4-f5b65382cfb6-kube-api-access-r59ps\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.400312 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:58.400235    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm podName:117720a7-6661-4079-81b4-f5b65382cfb6 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:58.900210256 +0000 UTC m=+64.372050085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7sgh7" (UID: "117720a7-6661-4079-81b4-f5b65382cfb6") : configmap "kube-rbac-proxy" not found
Sep 17 15:39:58.400312 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:58.400235    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:39:58.400312 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.400285    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/117720a7-6661-4079-81b4-f5b65382cfb6-data-volume\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.400487 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:58.400321    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls podName:117720a7-6661-4079-81b4-f5b65382cfb6 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:58.90030177 +0000 UTC m=+64.372141620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7sgh7" (UID: "117720a7-6661-4079-81b4-f5b65382cfb6") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:39:58.400487 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.400360    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/117720a7-6661-4079-81b4-f5b65382cfb6-crio-socket\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.400487 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.400461    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/117720a7-6661-4079-81b4-f5b65382cfb6-crio-socket\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.400683 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.400662    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/117720a7-6661-4079-81b4-f5b65382cfb6-data-volume\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.411778 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.411750    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59ps\" (UniqueName: \"kubernetes.io/projected/117720a7-6661-4079-81b4-f5b65382cfb6-kube-api-access-r59ps\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.904057 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.904028    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.904527 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:58.904097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:58.904527 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:58.904154    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:39:58.904527 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:58.904188    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:39:58.904527 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:58.904215    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm podName:117720a7-6661-4079-81b4-f5b65382cfb6 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:59.904200186 +0000 UTC m=+65.376040013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7sgh7" (UID: "117720a7-6661-4079-81b4-f5b65382cfb6") : configmap "kube-rbac-proxy" not found
Sep 17 15:39:58.904527 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:58.904230    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls podName:117720a7-6661-4079-81b4-f5b65382cfb6 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:59.904223666 +0000 UTC m=+65.376063490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7sgh7" (UID: "117720a7-6661-4079-81b4-f5b65382cfb6") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:39:59.476370 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:59.476329    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk" event={"ID":"57985d52-f04f-4487-84de-ddfd56e490af","Type":"ContainerStarted","Data":"b0fdce84f2cc3a3a93142b57b5264e98d985b87fd083b1b783c4eaacdf1e2e41"}
Sep 17 15:39:59.491850 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:59.491799    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk" podStartSLOduration=1.317558659 podStartE2EDuration="3.491782404s" totalImagesPullingTime="2.174223745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:59.491287513 +0000 UTC m=+64.963127359" watchObservedRunningTime="2026-09-17 15:39:59.491782404 +0000 UTC m=+64.963622250"
Sep 17 15:39:59.912304 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:59.912271    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:59.912699 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:39:59.912336    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:39:59.912699 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:59.912400    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:39:59.912699 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:59.912458    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:39:59.912699 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:59.912501    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm podName:117720a7-6661-4079-81b4-f5b65382cfb6 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:01.912468403 +0000 UTC m=+67.384308228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7sgh7" (UID: "117720a7-6661-4079-81b4-f5b65382cfb6") : configmap "kube-rbac-proxy" not found
Sep 17 15:39:59.912699 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:39:59.912551    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls podName:117720a7-6661-4079-81b4-f5b65382cfb6 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:01.912543032 +0000 UTC m=+67.384382860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7sgh7" (UID: "117720a7-6661-4079-81b4-f5b65382cfb6") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:00.013332 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:00.013284    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:40:00.013332 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:00.013332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:40:00.013553 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:00.013453    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:40:00.013553 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:00.013476    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle podName:1d1c6522-62ea-4c17-b992-33f236073899 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:08.013454011 +0000 UTC m=+73.485293853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle") pod "router-default-64b88459dd-xdnj5" (UID: "1d1c6522-62ea-4c17-b992-33f236073899") : configmap references non-existent config key: service-ca.crt
Sep 17 15:40:00.013553 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:00.013505    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs podName:1d1c6522-62ea-4c17-b992-33f236073899 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:08.013493418 +0000 UTC m=+73.485333242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs") pod "router-default-64b88459dd-xdnj5" (UID: "1d1c6522-62ea-4c17-b992-33f236073899") : secret "router-metrics-certs-default" not found
Sep 17 15:40:00.280022 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:00.279950    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vzk9l_9f8f3b74-61b5-4b22-bf59-ab2ec9054df1/dns-node-resolver/0.log"
Sep 17 15:40:00.634564 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:00.634532    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:00.707290 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:00.707257    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:00.820581 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:00.820543    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs\") pod \"network-metrics-daemon-c4llb\" (UID: \"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf\") " pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:40:00.820748 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:00.820635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:40:00.823125 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:00.823096    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:40:00.823125 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:00.823106    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:40:00.831352 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:00.831327    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 15:40:00.831462 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:00.831396    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs podName:e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf nodeName:}" failed. No retries permitted until 2026-09-17 15:41:04.831375839 +0000 UTC m=+130.303215669 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs") pod "network-metrics-daemon-c4llb" (UID: "e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf") : secret "metrics-daemon-secret" not found
Sep 17 15:40:00.833516 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:00.833489    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72f72846-2cc4-4793-a883-42382a787e30-original-pull-secret\") pod \"global-pull-secret-syncer-vt2l8\" (UID: \"72f72846-2cc4-4793-a883-42382a787e30\") " pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:40:00.922077 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:00.921983    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr\") pod \"network-check-target-twzfl\" (UID: \"dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7\") " pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:40:00.924671 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:00.924653    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:40:00.935259 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:00.935237    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:40:00.945513 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:00.945488    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7-kube-api-access-5b5fr\") pod \"network-check-target-twzfl\" (UID: \"dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7\") " pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:40:01.054753 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:01.054718    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:01.054961 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:01.054946    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xkd6d\""
Sep 17 15:40:01.062976 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:01.062953    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vt2l8"
Sep 17 15:40:01.064026 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:01.064009    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:40:01.156924 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:01.156860    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:01.284897 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:01.284815    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7845q_346c8e4c-4119-41a8-b798-c23d6de9b203/node-ca/0.log"
Sep 17 15:40:01.326224 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:01.326192    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:01.357339 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:01.357295    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vt2l8"]
Sep 17 15:40:01.361107 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:40:01.361082    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f72846_2cc4_4793_a883_42382a787e30.slice/crio-f33f3b0f262ac7fad8ab77b602300c5e84bc7960aa4adbf4605809f2d18f6e08 WatchSource:0}: Error finding container f33f3b0f262ac7fad8ab77b602300c5e84bc7960aa4adbf4605809f2d18f6e08: Status 404 returned error can't find the container with id f33f3b0f262ac7fad8ab77b602300c5e84bc7960aa4adbf4605809f2d18f6e08
Sep 17 15:40:01.375966 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:01.375941    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-twzfl"]
Sep 17 15:40:01.380238 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:40:01.380214    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6c2c61_5483_44b4_aeb9_e7ec2fc238f7.slice/crio-76b05063116a06f3255d6a3a4731fdbb8ebeba3b4f4ae54658efecc2fce26b99 WatchSource:0}: Error finding container 76b05063116a06f3255d6a3a4731fdbb8ebeba3b4f4ae54658efecc2fce26b99: Status 404 returned error can't find the container with id 76b05063116a06f3255d6a3a4731fdbb8ebeba3b4f4ae54658efecc2fce26b99
Sep 17 15:40:01.382612 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:01.382577    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:01.481993 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:01.481948    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vt2l8" event={"ID":"72f72846-2cc4-4793-a883-42382a787e30","Type":"ContainerStarted","Data":"f33f3b0f262ac7fad8ab77b602300c5e84bc7960aa4adbf4605809f2d18f6e08"}
Sep 17 15:40:01.483103 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:01.483079    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-twzfl" event={"ID":"dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7","Type":"ContainerStarted","Data":"76b05063116a06f3255d6a3a4731fdbb8ebeba3b4f4ae54658efecc2fce26b99"}
Sep 17 15:40:01.929107 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:01.929068    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:40:01.929607 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:01.929166    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:40:01.929607 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:01.929203    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:01.929607 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:01.929281    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:01.929607 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:01.929290    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm podName:117720a7-6661-4079-81b4-f5b65382cfb6 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:05.929272682 +0000 UTC m=+71.401112527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7sgh7" (UID: "117720a7-6661-4079-81b4-f5b65382cfb6") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:01.929607 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:01.929342    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls podName:117720a7-6661-4079-81b4-f5b65382cfb6 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:05.92932644 +0000 UTC m=+71.401166268 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7sgh7" (UID: "117720a7-6661-4079-81b4-f5b65382cfb6") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:02.001130 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.001092    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94596bc56-6f4rn"]
Sep 17 15:40:02.003336 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.003310    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94596bc56-6f4rn"
Sep 17 15:40:02.006234 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.006212    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 15:40:02.006234 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.006222    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 15:40:02.006822 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.006452    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 15:40:02.006822 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.006654    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-q6scv\""
Sep 17 15:40:02.006822 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.006815    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 15:40:02.007758 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.007735    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"]
Sep 17 15:40:02.009806 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.009589    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"
Sep 17 15:40:02.011414 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.011395    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 15:40:02.014257 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.014234    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94596bc56-6f4rn"]
Sep 17 15:40:02.019928 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.019901    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"]
Sep 17 15:40:02.030167 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.030137    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d4858ffc8-qx4gj\" (UID: \"1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"
Sep 17 15:40:02.030271 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.030184    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q2pg\" (UniqueName: \"kubernetes.io/projected/1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3-kube-api-access-2q2pg\") pod \"klusterlet-addon-workmgr-5d4858ffc8-qx4gj\" (UID: \"1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"
Sep 17 15:40:02.030271 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.030215    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3-tmp\") pod \"klusterlet-addon-workmgr-5d4858ffc8-qx4gj\" (UID: \"1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"
Sep 17 15:40:02.030368 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.030273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1917379-2b6b-4d26-b068-e1715c3c1cd4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-94596bc56-6f4rn\" (UID: \"b1917379-2b6b-4d26-b068-e1715c3c1cd4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94596bc56-6f4rn"
Sep 17 15:40:02.030368 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.030299    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd2tz\" (UniqueName: \"kubernetes.io/projected/b1917379-2b6b-4d26-b068-e1715c3c1cd4-kube-api-access-vd2tz\") pod \"managed-serviceaccount-addon-agent-94596bc56-6f4rn\" (UID: \"b1917379-2b6b-4d26-b068-e1715c3c1cd4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94596bc56-6f4rn"
Sep 17 15:40:02.080218 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.080182    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lpgls_02f7fe3e-05c4-49ac-97d2-35337336c9cb/serve-healthcheck-canary/0.log"
Sep 17 15:40:02.132465 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.131187    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d4858ffc8-qx4gj\" (UID: \"1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"
Sep 17 15:40:02.132465 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.131239    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2pg\" (UniqueName: \"kubernetes.io/projected/1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3-kube-api-access-2q2pg\") pod \"klusterlet-addon-workmgr-5d4858ffc8-qx4gj\" (UID: \"1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"
Sep 17 15:40:02.132465 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.131273    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3-tmp\") pod \"klusterlet-addon-workmgr-5d4858ffc8-qx4gj\" (UID: \"1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"
Sep 17 15:40:02.132465 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.131335    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1917379-2b6b-4d26-b068-e1715c3c1cd4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-94596bc56-6f4rn\" (UID: \"b1917379-2b6b-4d26-b068-e1715c3c1cd4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94596bc56-6f4rn"
Sep 17 15:40:02.132465 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.131362    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd2tz\" (UniqueName: \"kubernetes.io/projected/b1917379-2b6b-4d26-b068-e1715c3c1cd4-kube-api-access-vd2tz\") pod \"managed-serviceaccount-addon-agent-94596bc56-6f4rn\" (UID: \"b1917379-2b6b-4d26-b068-e1715c3c1cd4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94596bc56-6f4rn"
Sep 17 15:40:02.132920 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.132869    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3-tmp\") pod \"klusterlet-addon-workmgr-5d4858ffc8-qx4gj\" (UID: \"1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"
Sep 17 15:40:02.136018 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.135615    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1917379-2b6b-4d26-b068-e1715c3c1cd4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-94596bc56-6f4rn\" (UID: \"b1917379-2b6b-4d26-b068-e1715c3c1cd4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94596bc56-6f4rn"
Sep 17 15:40:02.136018 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.135975    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d4858ffc8-qx4gj\" (UID: \"1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"
Sep 17 15:40:02.140689 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.140646    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd2tz\" (UniqueName: \"kubernetes.io/projected/b1917379-2b6b-4d26-b068-e1715c3c1cd4-kube-api-access-vd2tz\") pod \"managed-serviceaccount-addon-agent-94596bc56-6f4rn\" (UID: \"b1917379-2b6b-4d26-b068-e1715c3c1cd4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94596bc56-6f4rn"
Sep 17 15:40:02.142878 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.142854    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q2pg\" (UniqueName: \"kubernetes.io/projected/1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3-kube-api-access-2q2pg\") pod \"klusterlet-addon-workmgr-5d4858ffc8-qx4gj\" (UID: \"1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"
Sep 17 15:40:02.232315 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.232202    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:40:02.232514 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:02.232475    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:40:02.232582 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:02.232540    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls podName:300fa6c7-126b-4872-bb0f-13ca5a4eca5c nodeName:}" failed. No retries permitted until 2026-09-17 15:40:34.232520612 +0000 UTC m=+99.704360451 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls") pod "dns-default-d7xgf" (UID: "300fa6c7-126b-4872-bb0f-13ca5a4eca5c") : secret "dns-default-metrics-tls" not found
Sep 17 15:40:02.326595 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.326551    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94596bc56-6f4rn"
Sep 17 15:40:02.331869 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.331674    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"
Sep 17 15:40:02.623124 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.623081    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94596bc56-6f4rn"]
Sep 17 15:40:02.628227 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:02.628087    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"]
Sep 17 15:40:03.046739 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:03.046658    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:03.109647 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:03.109612    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:03.492032 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:03.491964    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94596bc56-6f4rn" event={"ID":"b1917379-2b6b-4d26-b068-e1715c3c1cd4","Type":"ContainerStarted","Data":"d54a3ce0df618fcb86dbf97f44584d1ee74119cf57310dd83c3eb232ab4bf0e3"}
Sep 17 15:40:03.493923 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:03.493889    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj" event={"ID":"1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3","Type":"ContainerStarted","Data":"429c21a253ea275dc53651f81c1193a5f857c40820e18c6747be7ceee3b36349"}
Sep 17 15:40:03.929902 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:03.929729    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:05.963908 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:05.963870    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:40:05.964338 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:05.963956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:40:05.964338 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:05.964115    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:05.964338 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:05.964182    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls podName:117720a7-6661-4079-81b4-f5b65382cfb6 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:13.964161367 +0000 UTC m=+79.436001192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7sgh7" (UID: "117720a7-6661-4079-81b4-f5b65382cfb6") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:05.964542 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:05.964506    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:05.964596 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:05.964589    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm podName:117720a7-6661-4079-81b4-f5b65382cfb6 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:13.964561788 +0000 UTC m=+79.436401630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7sgh7" (UID: "117720a7-6661-4079-81b4-f5b65382cfb6") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:06.603918 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:06.603874    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:06.771411 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:06.771367    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:06.887342 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:06.887248    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:06.987748 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:06.987708    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:07.043672 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:07.043638    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:07.248971 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:07.248886    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:07.348841 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:07.348798    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:08.083257 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:08.083215    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:40:08.083681 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:08.083263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:40:08.083681 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:08.083405    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle podName:1d1c6522-62ea-4c17-b992-33f236073899 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:24.08338177 +0000 UTC m=+89.555221618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle") pod "router-default-64b88459dd-xdnj5" (UID: "1d1c6522-62ea-4c17-b992-33f236073899") : configmap references non-existent config key: service-ca.crt
Sep 17 15:40:08.083681 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:08.083432    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:40:08.083681 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:08.083495    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs podName:1d1c6522-62ea-4c17-b992-33f236073899 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:24.083480602 +0000 UTC m=+89.555320431 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs") pod "router-default-64b88459dd-xdnj5" (UID: "1d1c6522-62ea-4c17-b992-33f236073899") : secret "router-metrics-certs-default" not found
Sep 17 15:40:08.162494 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:08.162457    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:08.502092 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:08.502049    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:08.509635 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:08.509600    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vt2l8" event={"ID":"72f72846-2cc4-4793-a883-42382a787e30","Type":"ContainerStarted","Data":"66d8f0f4aa8b3d70649ad114b5b62abbd3e0ddd9faed4fb1927e2e7108b0d42b"}
Sep 17 15:40:08.511453 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:08.511405    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-twzfl" event={"ID":"dd6c2c61-5483-44b4-aeb9-e7ec2fc238f7","Type":"ContainerStarted","Data":"719f30b3138bb33393119665d638e6d74ee411be3dc0b9106e9bd8960697b6d0"}
Sep 17 15:40:08.511581 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:08.511491    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:40:08.513291 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:08.513255    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94596bc56-6f4rn" event={"ID":"b1917379-2b6b-4d26-b068-e1715c3c1cd4","Type":"ContainerStarted","Data":"6536a08cc735794e9caae90f99dae36a25044bc25ec299dec11dacc390e2efd5"}
Sep 17 15:40:08.525780 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:08.525727    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vt2l8" podStartSLOduration=66.357092642 podStartE2EDuration="1m12.525711554s" totalImagesPullingTime="6.168618912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:08.52465056 +0000 UTC m=+73.996490406" watchObservedRunningTime="2026-09-17 15:40:08.525711554 +0000 UTC m=+73.997551454"
Sep 17 15:40:08.539403 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:08.539350    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94596bc56-6f4rn" podStartSLOduration=2.6390774869999998 podStartE2EDuration="7.539332611s" totalImagesPullingTime="4.900255124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:08.538614484 +0000 UTC m=+74.010454343" watchObservedRunningTime="2026-09-17 15:40:08.539332611 +0000 UTC m=+74.011172457"
Sep 17 15:40:08.555104 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:08.555045    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-twzfl" podStartSLOduration=66.42433344 podStartE2EDuration="1m12.55502903s" totalImagesPullingTime="6.13069559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:08.553604487 +0000 UTC m=+74.025444333" watchObservedRunningTime="2026-09-17 15:40:08.55502903 +0000 UTC m=+74.026868881"
Sep 17 15:40:08.921948 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:08.921903    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:09.106655 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:09.106610    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:09.710107 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:09.710069    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:10.192595 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:10.192559    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:10.303936 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:10.303903    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:10.366668 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:10.366640    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:10.503355 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:10.503264    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:10.521535 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:10.521499    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj" event={"ID":"1028b2d6-6b8c-4bcb-9bf4-b0bc59c6a6f3","Type":"ContainerStarted","Data":"8573dc6eba03bcda442901ff52d2ab75184a9e907bba32934b836944590845ae"}
Sep 17 15:40:10.521765 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:10.521747    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"
Sep 17 15:40:10.523404 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:10.523386    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj"
Sep 17 15:40:10.538224 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:10.538180    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d4858ffc8-qx4gj" podStartSLOduration=2.563599027 podStartE2EDuration="9.538165086s" totalImagesPullingTime="6.974566059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:10.537112824 +0000 UTC m=+76.008952670" watchObservedRunningTime="2026-09-17 15:40:10.538165086 +0000 UTC m=+76.010004931"
Sep 17 15:40:10.706763 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:10.703957    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:11.107401 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:11.107357    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:11.164858 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:11.164824    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:11.711067 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:11.711029    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:12.310614 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:12.310567    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:12.515664 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:12.515628    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:12.710711 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:12.710666    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:12.826384 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:12.826352    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:13.115943 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:13.115913    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:13.505729 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:13.505638    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:14.034688 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:14.034649    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:40:14.035163 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:14.034812    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:40:14.035163 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:14.034827    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:14.035163 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:14.034914    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls podName:117720a7-6661-4079-81b4-f5b65382cfb6 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:30.034892665 +0000 UTC m=+95.506732502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7sgh7" (UID: "117720a7-6661-4079-81b4-f5b65382cfb6") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:14.035353 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:14.035334    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/117720a7-6661-4079-81b4-f5b65382cfb6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:40:14.107356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:14.107319    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:14.722321 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:14.722286    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:14.895036 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:14.894998    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:14.956712 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:14.956673    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:15.117075 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:15.117039    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:15.511943 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:15.511849    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:15.907392 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:15.907346    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:16.374085 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:16.374053    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:16.473780 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:16.473745    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:16.574472 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:16.574428    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:16.711551 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:16.711466    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:17.132067 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:17.132028    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:17.317154 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:17.317117    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:20.748977 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:20.748943    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:24.118367 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:24.118325    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:40:24.118367 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:24.118364    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:40:24.119158 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:24.119131    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1c6522-62ea-4c17-b992-33f236073899-service-ca-bundle\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:40:24.120641 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:24.120609    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d1c6522-62ea-4c17-b992-33f236073899-metrics-certs\") pod \"router-default-64b88459dd-xdnj5\" (UID: \"1d1c6522-62ea-4c17-b992-33f236073899\") " pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:40:24.313451 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:24.313420    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zswsd\""
Sep 17 15:40:24.321672 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:24.321648    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:40:24.593547 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:24.593516    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64b88459dd-xdnj5"]
Sep 17 15:40:24.596488 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:40:24.596428    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1c6522_62ea_4c17_b992_33f236073899.slice/crio-feb799c4a5c2756b1845c1939f6c955a4d1a63ecb92b7d735f38a2f095c803fd WatchSource:0}: Error finding container feb799c4a5c2756b1845c1939f6c955a4d1a63ecb92b7d735f38a2f095c803fd: Status 404 returned error can't find the container with id feb799c4a5c2756b1845c1939f6c955a4d1a63ecb92b7d735f38a2f095c803fd
Sep 17 15:40:25.566344 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:25.566313    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64b88459dd-xdnj5" event={"ID":"1d1c6522-62ea-4c17-b992-33f236073899","Type":"ContainerStarted","Data":"77cc608888cf9a5408af1d2c406d8f092ba1ff3e2a847f97b3b956d6475fe65f"}
Sep 17 15:40:25.566748 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:25.566358    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64b88459dd-xdnj5" event={"ID":"1d1c6522-62ea-4c17-b992-33f236073899","Type":"ContainerStarted","Data":"feb799c4a5c2756b1845c1939f6c955a4d1a63ecb92b7d735f38a2f095c803fd"}
Sep 17 15:40:25.588567 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:25.588507    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-64b88459dd-xdnj5" podStartSLOduration=33.588487224 podStartE2EDuration="33.588487224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:25.587942834 +0000 UTC m=+91.059782818" watchObservedRunningTime="2026-09-17 15:40:25.588487224 +0000 UTC m=+91.060327073"
Sep 17 15:40:26.322767 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:26.322727    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:40:26.325106 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:26.325080    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:40:26.569405 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:26.569375    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:40:26.570643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:26.570620    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-64b88459dd-xdnj5"
Sep 17 15:40:27.517758 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:27.517717    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:27.673592 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:27.673554    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:27.991873 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:27.991833    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:30.061173 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:30.061133    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:40:30.063434 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:30.063407    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/117720a7-6661-4079-81b4-f5b65382cfb6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7sgh7\" (UID: \"117720a7-6661-4079-81b4-f5b65382cfb6\") " pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:40:30.253326 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:30.253285    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7sgh7"
Sep 17 15:40:30.533605 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:30.533572    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7sgh7"]
Sep 17 15:40:30.536793 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:40:30.536768    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117720a7_6661_4079_81b4_f5b65382cfb6.slice/crio-4751752669e1fcac7d9050a8b6f98ffa226edb896f96ee94545dcfe2195cb974 WatchSource:0}: Error finding container 4751752669e1fcac7d9050a8b6f98ffa226edb896f96ee94545dcfe2195cb974: Status 404 returned error can't find the container with id 4751752669e1fcac7d9050a8b6f98ffa226edb896f96ee94545dcfe2195cb974
Sep 17 15:40:30.583456 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:30.583411    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7sgh7" event={"ID":"117720a7-6661-4079-81b4-f5b65382cfb6","Type":"ContainerStarted","Data":"4751752669e1fcac7d9050a8b6f98ffa226edb896f96ee94545dcfe2195cb974"}
Sep 17 15:40:31.590743 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:31.590665    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7sgh7" event={"ID":"117720a7-6661-4079-81b4-f5b65382cfb6","Type":"ContainerStarted","Data":"533624f8b202114a85ca26daba1fe88d3e80f5697413442a77237f9eec81fd58"}
Sep 17 15:40:31.590743 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:31.590732    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7sgh7" event={"ID":"117720a7-6661-4079-81b4-f5b65382cfb6","Type":"ContainerStarted","Data":"7088a9ca378e855b6fd89262eb7c7a9a267e0364a35be9de0ec79b481610142f"}
Sep 17 15:40:32.595981 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:32.595952    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7sgh7" event={"ID":"117720a7-6661-4079-81b4-f5b65382cfb6","Type":"ContainerStarted","Data":"de5714821af5aacbbb8dbcee68d0c9e157305c1126c340a3b1500842511992de"}
Sep 17 15:40:32.626978 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:32.626884    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7sgh7" podStartSLOduration=32.903511262 podStartE2EDuration="34.62686911s" totalImagesPullingTime="1.723357848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:32.626050274 +0000 UTC m=+98.097890120" watchObservedRunningTime="2026-09-17 15:40:32.62686911 +0000 UTC m=+98.098708956"
Sep 17 15:40:32.899068 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:32.898991    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:32.989972 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:32.989935    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:34.291905 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:34.291871    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:40:34.294140 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:34.294106    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/300fa6c7-126b-4872-bb0f-13ca5a4eca5c-metrics-tls\") pod \"dns-default-d7xgf\" (UID: \"300fa6c7-126b-4872-bb0f-13ca5a4eca5c\") " pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:40:34.582874 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:34.582787    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9xnvp\""
Sep 17 15:40:34.591524 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:34.591502    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:40:34.864875 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:34.864841    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d7xgf"]
Sep 17 15:40:34.867722 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:40:34.867688    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300fa6c7_126b_4872_bb0f_13ca5a4eca5c.slice/crio-d89ea4c67ffa53fbcce42b954451dba96076b821e4f485a0a5aa106d01eaea73 WatchSource:0}: Error finding container d89ea4c67ffa53fbcce42b954451dba96076b821e4f485a0a5aa106d01eaea73: Status 404 returned error can't find the container with id d89ea4c67ffa53fbcce42b954451dba96076b821e4f485a0a5aa106d01eaea73
Sep 17 15:40:35.284186 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.284094    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:35.384310 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.384270    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:35.609304 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.609266    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7xgf" event={"ID":"300fa6c7-126b-4872-bb0f-13ca5a4eca5c","Type":"ContainerStarted","Data":"d89ea4c67ffa53fbcce42b954451dba96076b821e4f485a0a5aa106d01eaea73"}
Sep 17 15:40:35.727159 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.727114    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:35.783654 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.783620    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"]
Sep 17 15:40:35.786848 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.786825    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:35.790281 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.790252    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 15:40:35.790660 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.790626    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 15:40:35.791404 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.791341    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 15:40:35.791404 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.791351    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7wzh7\""
Sep 17 15:40:35.791643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.791419    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 15:40:35.791643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.791489    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 15:40:35.796633 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.796610    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"]
Sep 17 15:40:35.802460 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.802424    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:35.901661 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.901581    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f463991b-2e30-48fb-9007-7a6a4fa6a001-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-qbc2c\" (UID: \"f463991b-2e30-48fb-9007-7a6a4fa6a001\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:35.901661 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.901627    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f463991b-2e30-48fb-9007-7a6a4fa6a001-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-qbc2c\" (UID: \"f463991b-2e30-48fb-9007-7a6a4fa6a001\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:35.901661 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.901646    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnhhx\" (UniqueName: \"kubernetes.io/projected/f463991b-2e30-48fb-9007-7a6a4fa6a001-kube-api-access-bnhhx\") pod \"prometheus-operator-68bcd7f987-qbc2c\" (UID: \"f463991b-2e30-48fb-9007-7a6a4fa6a001\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:35.901905 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:35.901774    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f463991b-2e30-48fb-9007-7a6a4fa6a001-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-qbc2c\" (UID: \"f463991b-2e30-48fb-9007-7a6a4fa6a001\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:36.002534 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.002496    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f463991b-2e30-48fb-9007-7a6a4fa6a001-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-qbc2c\" (UID: \"f463991b-2e30-48fb-9007-7a6a4fa6a001\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:36.002729 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.002564    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f463991b-2e30-48fb-9007-7a6a4fa6a001-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-qbc2c\" (UID: \"f463991b-2e30-48fb-9007-7a6a4fa6a001\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:36.002729 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.002603    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f463991b-2e30-48fb-9007-7a6a4fa6a001-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-qbc2c\" (UID: \"f463991b-2e30-48fb-9007-7a6a4fa6a001\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:36.002729 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.002627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnhhx\" (UniqueName: \"kubernetes.io/projected/f463991b-2e30-48fb-9007-7a6a4fa6a001-kube-api-access-bnhhx\") pod \"prometheus-operator-68bcd7f987-qbc2c\" (UID: \"f463991b-2e30-48fb-9007-7a6a4fa6a001\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:36.002729 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:36.002667    2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 15:40:36.002928 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:36.002757    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f463991b-2e30-48fb-9007-7a6a4fa6a001-prometheus-operator-tls podName:f463991b-2e30-48fb-9007-7a6a4fa6a001 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:36.502734213 +0000 UTC m=+101.974574043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/f463991b-2e30-48fb-9007-7a6a4fa6a001-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-qbc2c" (UID: "f463991b-2e30-48fb-9007-7a6a4fa6a001") : secret "prometheus-operator-tls" not found
Sep 17 15:40:36.003385 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.003358    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f463991b-2e30-48fb-9007-7a6a4fa6a001-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-qbc2c\" (UID: \"f463991b-2e30-48fb-9007-7a6a4fa6a001\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:36.005260 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.005237    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f463991b-2e30-48fb-9007-7a6a4fa6a001-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-qbc2c\" (UID: \"f463991b-2e30-48fb-9007-7a6a4fa6a001\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:36.014841 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.014811    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnhhx\" (UniqueName: \"kubernetes.io/projected/f463991b-2e30-48fb-9007-7a6a4fa6a001-kube-api-access-bnhhx\") pod \"prometheus-operator-68bcd7f987-qbc2c\" (UID: \"f463991b-2e30-48fb-9007-7a6a4fa6a001\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:36.213138 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.213035    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:36.507351 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.507328    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f463991b-2e30-48fb-9007-7a6a4fa6a001-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-qbc2c\" (UID: \"f463991b-2e30-48fb-9007-7a6a4fa6a001\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:36.509629 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.509609    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f463991b-2e30-48fb-9007-7a6a4fa6a001-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-qbc2c\" (UID: \"f463991b-2e30-48fb-9007-7a6a4fa6a001\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:36.618372 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.618339    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7xgf" event={"ID":"300fa6c7-126b-4872-bb0f-13ca5a4eca5c","Type":"ContainerStarted","Data":"984377d9be20eed6c74b1d8730fce1c86be06356516d6cd63256c13ef062cdb3"}
Sep 17 15:40:36.618566 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.618410    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7xgf" event={"ID":"300fa6c7-126b-4872-bb0f-13ca5a4eca5c","Type":"ContainerStarted","Data":"b624d7ec1d0f75fa18f601b7c3eda207378c00cce754fb2b78d967dd68b446c7"}
Sep 17 15:40:36.618566 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.618482    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:40:36.636001 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.635949    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d7xgf" podStartSLOduration=65.17626031 podStartE2EDuration="1m6.635934118s" totalImagesPullingTime="1.459673808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:36.635303835 +0000 UTC m=+102.107143680" watchObservedRunningTime="2026-09-17 15:40:36.635934118 +0000 UTC m=+102.107773967"
Sep 17 15:40:36.699134 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.699093    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"
Sep 17 15:40:36.970129 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:36.970095    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c"]
Sep 17 15:40:36.973310 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:40:36.973286    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf463991b_2e30_48fb_9007_7a6a4fa6a001.slice/crio-402cf191b3d7ee49036f2df07d4ac218f384097e7704827e67d84e74c5492b7e WatchSource:0}: Error finding container 402cf191b3d7ee49036f2df07d4ac218f384097e7704827e67d84e74c5492b7e: Status 404 returned error can't find the container with id 402cf191b3d7ee49036f2df07d4ac218f384097e7704827e67d84e74c5492b7e
Sep 17 15:40:37.495152 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:37.495118    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:37.590456 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:37.590359    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:37.623916 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:37.623882    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c" event={"ID":"f463991b-2e30-48fb-9007-7a6a4fa6a001","Type":"ContainerStarted","Data":"402cf191b3d7ee49036f2df07d4ac218f384097e7704827e67d84e74c5492b7e"}
Sep 17 15:40:37.781193 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:37.781091    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:38.163398 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:38.163365    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:38.264470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:38.264426    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:38.486118 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:38.486007    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:38.587103 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:38.587061    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:38.628322 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:38.628290    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c" event={"ID":"f463991b-2e30-48fb-9007-7a6a4fa6a001","Type":"ContainerStarted","Data":"512d42db3614eaa53908a85def1ab984cd73a90e7ffa1a7020f9e9cefee12865"}
Sep 17 15:40:38.628714 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:38.628338    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c" event={"ID":"f463991b-2e30-48fb-9007-7a6a4fa6a001","Type":"ContainerStarted","Data":"dc52e304e35c29c925db9d4f9bdb204e793d1245eb0f8a9656eb2492431335d1"}
Sep 17 15:40:38.649629 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:38.649579    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qbc2c" podStartSLOduration=2.41564801 podStartE2EDuration="3.649563629s" totalImagesPullingTime="1.233915619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:38.648476345 +0000 UTC m=+104.120316181" watchObservedRunningTime="2026-09-17 15:40:38.649563629 +0000 UTC m=+104.121403474"
Sep 17 15:40:38.653148 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:38.653127    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:38.968457 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:38.968413    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:39.036314 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:39.036283    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:39.171553 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:39.171515    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:39.371007 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:39.370978    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:39.519128 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:39.519099    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-twzfl"
Sep 17 15:40:39.654076 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:39.653986    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:39.865769 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:39.865737    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:40.118873 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.118831    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:40.140654 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.140625    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rnt5p"]
Sep 17 15:40:40.175435 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.175407    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.177551 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.177524    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 15:40:40.177691 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.177525    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m94zj\""
Sep 17 15:40:40.177784 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.177771    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 15:40:40.179965 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.179947    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 15:40:40.204987 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.204953    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:40.267256 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.267218    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:40.338592 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.338551    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-metrics-client-ca\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.338592 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.338596    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-root\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.338813 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.338620    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-sys\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.338813 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.338741    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqh2v\" (UniqueName: \"kubernetes.io/projected/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-kube-api-access-vqh2v\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.338813 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.338775    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-textfile\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.338813 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.338795    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.338944 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.338856    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-tls\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.338944 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.338873    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-accelerators-collector-config\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.338944 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.338896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-wtmp\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440045 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.439960    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-tls\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440045 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.439998    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-accelerators-collector-config\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440045 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.440021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-wtmp\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440045 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.440045    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-metrics-client-ca\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440354 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.440070    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-root\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440354 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.440102    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-sys\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440354 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:40.440125    2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 15:40:40.440354 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.440163    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqh2v\" (UniqueName: \"kubernetes.io/projected/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-kube-api-access-vqh2v\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440354 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:40:40.440202    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-tls podName:25f51d67-a0da-4d03-8fa9-9149fcdbbb92 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:40.940181212 +0000 UTC m=+106.412021046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-tls") pod "node-exporter-rnt5p" (UID: "25f51d67-a0da-4d03-8fa9-9149fcdbbb92") : secret "node-exporter-tls" not found
Sep 17 15:40:40.440354 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.440212    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-root\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440354 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.440235    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-textfile\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440354 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.440251    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-sys\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440354 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.440213    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-wtmp\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440354 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.440267    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440758 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.440544    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-textfile\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440758 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.440647    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-accelerators-collector-config\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.440758 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.440673    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-metrics-client-ca\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.442518 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.442496    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.449694 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.449670    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqh2v\" (UniqueName: \"kubernetes.io/projected/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-kube-api-access-vqh2v\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.452403 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.452375    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:40.552617 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.552577    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:40.651858 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.651790    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:40.802765 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.802673    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:40.907210 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.907156    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:40.944550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.944519    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-tls\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:40.946900 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:40.946870    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/25f51d67-a0da-4d03-8fa9-9149fcdbbb92-node-exporter-tls\") pod \"node-exporter-rnt5p\" (UID: \"25f51d67-a0da-4d03-8fa9-9149fcdbbb92\") " pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:41.084990 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.084907    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rnt5p"
Sep 17 15:40:41.118300 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.118262    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:41.127764 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:40:41.127735    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f51d67_a0da_4d03_8fa9_9149fcdbbb92.slice/crio-1ab54e0295eb2d5f757a858609a16743e3b454ff79b95ba8f704e604d6ad90e0 WatchSource:0}: Error finding container 1ab54e0295eb2d5f757a858609a16743e3b454ff79b95ba8f704e604d6ad90e0: Status 404 returned error can't find the container with id 1ab54e0295eb2d5f757a858609a16743e3b454ff79b95ba8f704e604d6ad90e0
Sep 17 15:40:41.198897 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.198861    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:41.210033 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.210002    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:40:41.245878 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.245850    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:40:41.246043 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.246022    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.249328 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.249307    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 15:40:41.249498 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.249341    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 15:40:41.249498 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.249311    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 15:40:41.249621 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.249609    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 15:40:41.249621 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.249628    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 15:40:41.249846 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.249833    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9lrw6\""
Sep 17 15:40:41.250102 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.250055    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 15:40:41.250356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.250336    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 15:40:41.250614 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.250597    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 15:40:41.250891 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.250873    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 15:40:41.280648 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.280607    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:41.346642 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.346559    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/029afaa8-e959-441d-a9e0-6c80551fb465-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.346642 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.346599    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.346642 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.346638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.346872 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.346682    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-web-config\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.346872 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.346705    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.346872 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.346722    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/029afaa8-e959-441d-a9e0-6c80551fb465-tls-assets\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.346872 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.346742    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.346872 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.346757    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/029afaa8-e959-441d-a9e0-6c80551fb465-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.346872 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.346807    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8njn\" (UniqueName: \"kubernetes.io/projected/029afaa8-e959-441d-a9e0-6c80551fb465-kube-api-access-m8njn\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.346872 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.346839    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-config-volume\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.347118 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.346888    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/029afaa8-e959-441d-a9e0-6c80551fb465-config-out\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.347118 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.346927    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029afaa8-e959-441d-a9e0-6c80551fb465-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.347118 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.346950    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.392001 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.391966    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:41.447614 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.447576    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-web-config\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.447614 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.447610    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.447862 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.447630    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/029afaa8-e959-441d-a9e0-6c80551fb465-tls-assets\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.447862 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.447648    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.447862 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.447664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/029afaa8-e959-441d-a9e0-6c80551fb465-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.447862 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.447682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8njn\" (UniqueName: \"kubernetes.io/projected/029afaa8-e959-441d-a9e0-6c80551fb465-kube-api-access-m8njn\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.447862 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.447704    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-config-volume\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.447862 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.447739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/029afaa8-e959-441d-a9e0-6c80551fb465-config-out\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.447862 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.447778    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029afaa8-e959-441d-a9e0-6c80551fb465-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.447862 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.447804    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.447862 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.447845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/029afaa8-e959-441d-a9e0-6c80551fb465-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.448317 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.447875    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.448317 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.447913    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.448317 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.448181    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/029afaa8-e959-441d-a9e0-6c80551fb465-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.449122 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.449091    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029afaa8-e959-441d-a9e0-6c80551fb465-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.449122 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.449112    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/029afaa8-e959-441d-a9e0-6c80551fb465-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.451010 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.450984    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.451116 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.451067    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-config-volume\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.451250 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.451226    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.451549 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.451527    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.451653 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.451622    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.451868 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.451845    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-web-config\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.452306 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.452290    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.452351 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.452290    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/029afaa8-e959-441d-a9e0-6c80551fb465-tls-assets\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.452868 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.452847    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/029afaa8-e959-441d-a9e0-6c80551fb465-config-out\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.458743 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.458724    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8njn\" (UniqueName: \"kubernetes.io/projected/029afaa8-e959-441d-a9e0-6c80551fb465-kube-api-access-m8njn\") pod \"alertmanager-main-0\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.555859 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.555820    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.644320 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.644247    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rnt5p" event={"ID":"25f51d67-a0da-4d03-8fa9-9149fcdbbb92","Type":"ContainerStarted","Data":"1ab54e0295eb2d5f757a858609a16743e3b454ff79b95ba8f704e604d6ad90e0"}
Sep 17 15:40:41.771650 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.771611    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:41.834459 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.834406    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:41.860231 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:41.860196    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:40:41.863518 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:40:41.863493    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029afaa8_e959_441d_a9e0_6c80551fb465.slice/crio-a1f920d63d373a1ff4c9a1fe27f342ddef2208b3677d1113de9a22a8afe80a3d WatchSource:0}: Error finding container a1f920d63d373a1ff4c9a1fe27f342ddef2208b3677d1113de9a22a8afe80a3d: Status 404 returned error can't find the container with id a1f920d63d373a1ff4c9a1fe27f342ddef2208b3677d1113de9a22a8afe80a3d
Sep 17 15:40:42.221297 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.219850    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7948667c77-xgmc9"]
Sep 17 15:40:42.240835 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.240804    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7948667c77-xgmc9"]
Sep 17 15:40:42.240989 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.240954    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.243917 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.243599    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 15:40:42.243917 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.243643    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 15:40:42.243917 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.243661    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2bceajetbuhip\""
Sep 17 15:40:42.243917 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.243672    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 15:40:42.243917 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.243734    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 15:40:42.243917 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.243643    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pcnfj\""
Sep 17 15:40:42.244302 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.244224    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 15:40:42.353654 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.353621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3925522f-d646-4bec-b1c1-58d3383380df-metrics-client-ca\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.353654 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.353660    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.353885 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.353684    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.353885 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.353723    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frtl4\" (UniqueName: \"kubernetes.io/projected/3925522f-d646-4bec-b1c1-58d3383380df-kube-api-access-frtl4\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.353885 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.353750    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-tls\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.353885 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.353766    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.354036 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.353888    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-grpc-tls\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.354036 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.353934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.455254 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.455223    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-grpc-tls\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.455419 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.455266    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.455419 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.455392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3925522f-d646-4bec-b1c1-58d3383380df-metrics-client-ca\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.455535 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.455465    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.455535 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.455493    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.455535 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.455513    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtl4\" (UniqueName: \"kubernetes.io/projected/3925522f-d646-4bec-b1c1-58d3383380df-kube-api-access-frtl4\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.455677 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.455544    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-tls\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.455677 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.455562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.456333 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.456296    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3925522f-d646-4bec-b1c1-58d3383380df-metrics-client-ca\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.458085 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.458041    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.458353 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.458319    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-grpc-tls\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.458454 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.458420    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.458499 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.458421    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.458499 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.458458    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-tls\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.458575 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.458515    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3925522f-d646-4bec-b1c1-58d3383380df-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.463831 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.463804    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtl4\" (UniqueName: \"kubernetes.io/projected/3925522f-d646-4bec-b1c1-58d3383380df-kube-api-access-frtl4\") pod \"thanos-querier-7948667c77-xgmc9\" (UID: \"3925522f-d646-4bec-b1c1-58d3383380df\") " pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.552615 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.552535    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:42.648476 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.648419    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerStarted","Data":"a1f920d63d373a1ff4c9a1fe27f342ddef2208b3677d1113de9a22a8afe80a3d"}
Sep 17 15:40:42.933101 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:42.933039    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7948667c77-xgmc9"]
Sep 17 15:40:42.941516 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:40:42.941489    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3925522f_d646_4bec_b1c1_58d3383380df.slice/crio-480df9b005fe6f4d7fd3e42ac37ac71ef37a91db70d2beba5b84c4a47ff0465c WatchSource:0}: Error finding container 480df9b005fe6f4d7fd3e42ac37ac71ef37a91db70d2beba5b84c4a47ff0465c: Status 404 returned error can't find the container with id 480df9b005fe6f4d7fd3e42ac37ac71ef37a91db70d2beba5b84c4a47ff0465c
Sep 17 15:40:43.076656 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.076620    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:43.177154 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.177071    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:43.475339 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.475306    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:43.653825 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.653787    2574 generic.go:356] "Generic (PLEG): container finished" podID="25f51d67-a0da-4d03-8fa9-9149fcdbbb92" containerID="fed2a816bd425b99aa572d909aee4511ea097a155ea501eb2bc356ef98286e8e" exitCode=0
Sep 17 15:40:43.653973 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.653876    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rnt5p" event={"ID":"25f51d67-a0da-4d03-8fa9-9149fcdbbb92","Type":"ContainerDied","Data":"fed2a816bd425b99aa572d909aee4511ea097a155ea501eb2bc356ef98286e8e"}
Sep 17 15:40:43.655191 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.655166    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9" event={"ID":"3925522f-d646-4bec-b1c1-58d3383380df","Type":"ContainerStarted","Data":"480df9b005fe6f4d7fd3e42ac37ac71ef37a91db70d2beba5b84c4a47ff0465c"}
Sep 17 15:40:43.679229 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.679201    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:43.776677 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.776653    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5885cd8684-22xxn"]
Sep 17 15:40:43.807314 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.807218    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5885cd8684-22xxn"]
Sep 17 15:40:43.807314 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.807283    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:43.810344 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.810318    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 15:40:43.810493 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.810365    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 15:40:43.810493 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.810381    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t7h6z\""
Sep 17 15:40:43.810493 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.810318    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 15:40:43.810493 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.810327    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 15:40:43.810712 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.810318    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 15:40:43.810712 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.810610    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 15:40:43.810712 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.810613    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 15:40:43.968718 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.968681    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-config\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:43.968718 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.968726    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-oauth-serving-cert\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:43.969257 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.968806    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-oauth-config\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:43.969257 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.968881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-286x2\" (UniqueName: \"kubernetes.io/projected/eb0eb4cb-25c7-4040-98f4-677d4677598c-kube-api-access-286x2\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:43.969257 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.968936    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-serving-cert\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:43.969257 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:43.968969    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-service-ca\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:44.070509 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.070405    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-serving-cert\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:44.070509 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.070469    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-service-ca\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:44.070728 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.070531    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-config\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:44.070728 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.070545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-oauth-serving-cert\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:44.070728 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.070592    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-oauth-config\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:44.070728 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.070616    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-286x2\" (UniqueName: \"kubernetes.io/projected/eb0eb4cb-25c7-4040-98f4-677d4677598c-kube-api-access-286x2\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:44.071687 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.071658    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-oauth-serving-cert\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:44.071829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.071662    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-config\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:44.071829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.071728    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-service-ca\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:44.073678 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.073654    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-serving-cert\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:44.073774 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.073762    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-oauth-config\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:44.083231 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.083212    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-286x2\" (UniqueName: \"kubernetes.io/projected/eb0eb4cb-25c7-4040-98f4-677d4677598c-kube-api-access-286x2\") pod \"console-5885cd8684-22xxn\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") " pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:44.120041 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.120005    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:44.429518 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.429485    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5885cd8684-22xxn"]
Sep 17 15:40:44.432853 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:40:44.432828    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0eb4cb_25c7_4040_98f4_677d4677598c.slice/crio-e6610502443a8c99a461842c08e042def3cd94e10851c1ffa5aaa892c970d81f WatchSource:0}: Error finding container e6610502443a8c99a461842c08e042def3cd94e10851c1ffa5aaa892c970d81f: Status 404 returned error can't find the container with id e6610502443a8c99a461842c08e042def3cd94e10851c1ffa5aaa892c970d81f
Sep 17 15:40:44.544016 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.543966    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7f9446879-llkrn"]
Sep 17 15:40:44.555627 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.555586    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.557798 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.557748    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f9446879-llkrn"]
Sep 17 15:40:44.557966 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.557943    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 15:40:44.558067 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.558012    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9t84kkkhvfpe7\""
Sep 17 15:40:44.558123 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.558105    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 15:40:44.558370 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.558353    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-49msw\""
Sep 17 15:40:44.558370 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.558363    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 15:40:44.558521 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.558378    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 15:40:44.661294 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.661255    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rnt5p" event={"ID":"25f51d67-a0da-4d03-8fa9-9149fcdbbb92","Type":"ContainerStarted","Data":"9d95d3a060df85430eaaa9b836c08f5ce6a899ad36fdb30e728249a2b4b4ce9d"}
Sep 17 15:40:44.661499 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.661318    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rnt5p" event={"ID":"25f51d67-a0da-4d03-8fa9-9149fcdbbb92","Type":"ContainerStarted","Data":"3d8c16d468f93fce522a9e1456d17d5aaa878befc7ce9b66e138981882faeb95"}
Sep 17 15:40:44.663204 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.663174    2574 generic.go:356] "Generic (PLEG): container finished" podID="029afaa8-e959-441d-a9e0-6c80551fb465" containerID="67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063" exitCode=0
Sep 17 15:40:44.663315 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.663291    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerDied","Data":"67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063"}
Sep 17 15:40:44.664762 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.664734    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5885cd8684-22xxn" event={"ID":"eb0eb4cb-25c7-4040-98f4-677d4677598c","Type":"ContainerStarted","Data":"e6610502443a8c99a461842c08e042def3cd94e10851c1ffa5aaa892c970d81f"}
Sep 17 15:40:44.675349 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.675318    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1191d1d7-63b6-43db-9065-9103a0195353-secret-metrics-server-client-certs\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.675465 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.675357    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1191d1d7-63b6-43db-9065-9103a0195353-audit-log\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.675465 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.675413    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1191d1d7-63b6-43db-9065-9103a0195353-metrics-server-audit-profiles\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.675580 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.675474    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1191d1d7-63b6-43db-9065-9103a0195353-secret-metrics-server-tls\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.675580 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.675510    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nts78\" (UniqueName: \"kubernetes.io/projected/1191d1d7-63b6-43db-9065-9103a0195353-kube-api-access-nts78\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.675580 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.675549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1191d1d7-63b6-43db-9065-9103a0195353-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.675580 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.675571    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1191d1d7-63b6-43db-9065-9103a0195353-client-ca-bundle\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.683275 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.683196    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rnt5p" podStartSLOduration=3.063171631 podStartE2EDuration="4.683183063s" totalImagesPullingTime="1.527860647s" totalInitContainerRuntime="92.150785ms" isStatefulPod=true podCreationTimestamp="2026-09-17 15:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:44.681978886 +0000 UTC m=+110.153818730" watchObservedRunningTime="2026-09-17 15:40:44.683183063 +0000 UTC m=+110.155022909"
Sep 17 15:40:44.778707 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.778668    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1191d1d7-63b6-43db-9065-9103a0195353-secret-metrics-server-tls\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.779061 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.779034    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nts78\" (UniqueName: \"kubernetes.io/projected/1191d1d7-63b6-43db-9065-9103a0195353-kube-api-access-nts78\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.779198 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.779127    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1191d1d7-63b6-43db-9065-9103a0195353-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.779198 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.779158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1191d1d7-63b6-43db-9065-9103a0195353-client-ca-bundle\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.779330 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.779229    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1191d1d7-63b6-43db-9065-9103a0195353-secret-metrics-server-client-certs\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.779330 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.779254    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1191d1d7-63b6-43db-9065-9103a0195353-audit-log\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.779608 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.779580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1191d1d7-63b6-43db-9065-9103a0195353-metrics-server-audit-profiles\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.780180 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.779961    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1191d1d7-63b6-43db-9065-9103a0195353-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.780325 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.780299    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1191d1d7-63b6-43db-9065-9103a0195353-audit-log\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.780552 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.780529    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1191d1d7-63b6-43db-9065-9103a0195353-metrics-server-audit-profiles\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.782257 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.782233    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1191d1d7-63b6-43db-9065-9103a0195353-secret-metrics-server-tls\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.782466 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.782433    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1191d1d7-63b6-43db-9065-9103a0195353-secret-metrics-server-client-certs\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.782563 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.782542    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1191d1d7-63b6-43db-9065-9103a0195353-client-ca-bundle\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.788242 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.788219    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nts78\" (UniqueName: \"kubernetes.io/projected/1191d1d7-63b6-43db-9065-9103a0195353-kube-api-access-nts78\") pod \"metrics-server-7f9446879-llkrn\" (UID: \"1191d1d7-63b6-43db-9065-9103a0195353\") " pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.866236 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.866199    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:40:44.888608 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.888570    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:44.954806 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:44.954717    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:45.210777 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:45.210695    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:45.386789 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:45.386759    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f9446879-llkrn"]
Sep 17 15:40:45.429978 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:40:45.429948    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1191d1d7_63b6_43db_9065_9103a0195353.slice/crio-568c70282fca9b44e3c051715cf15b5a822743f02a4abc33d0cf3179e160f925 WatchSource:0}: Error finding container 568c70282fca9b44e3c051715cf15b5a822743f02a4abc33d0cf3179e160f925: Status 404 returned error can't find the container with id 568c70282fca9b44e3c051715cf15b5a822743f02a4abc33d0cf3179e160f925
Sep 17 15:40:45.671723 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:45.671688    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f9446879-llkrn" event={"ID":"1191d1d7-63b6-43db-9065-9103a0195353","Type":"ContainerStarted","Data":"568c70282fca9b44e3c051715cf15b5a822743f02a4abc33d0cf3179e160f925"}
Sep 17 15:40:45.674114 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:45.674083    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9" event={"ID":"3925522f-d646-4bec-b1c1-58d3383380df","Type":"ContainerStarted","Data":"afc331719eada5b3649a1f5ead71df701bf990f2f25abf25abbe5faf5e011b30"}
Sep 17 15:40:46.439947 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:46.439907    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:46.518860 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:46.518822    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:46.626528 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:46.626492    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d7xgf"
Sep 17 15:40:46.684037 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:46.683980    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9" event={"ID":"3925522f-d646-4bec-b1c1-58d3383380df","Type":"ContainerStarted","Data":"d12d7e46ebc3e449e2ffc4e0cda15012256c427ebef9429d751d580ef0afb27f"}
Sep 17 15:40:46.684231 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:46.684072    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9" event={"ID":"3925522f-d646-4bec-b1c1-58d3383380df","Type":"ContainerStarted","Data":"39b8bf5414a8c50d89c312ca03ad1e87df0a2739d76befcdb3bee4a46a265b32"}
Sep 17 15:40:48.695084 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:48.695048    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerStarted","Data":"ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f"}
Sep 17 15:40:48.695340 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:48.695146    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerStarted","Data":"20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6"}
Sep 17 15:40:48.701694 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:48.701636    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5885cd8684-22xxn" event={"ID":"eb0eb4cb-25c7-4040-98f4-677d4677598c","Type":"ContainerStarted","Data":"7798773414eebf8a837e80dca98e0d8f386e8f7697cf6bc8a3c7217b9a1772a5"}
Sep 17 15:40:48.704071 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:48.704034    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f9446879-llkrn" event={"ID":"1191d1d7-63b6-43db-9065-9103a0195353","Type":"ContainerStarted","Data":"246d2398685de560becc52d5f1379eb6376c9be603250a2639ce5b7dc799e577"}
Sep 17 15:40:48.710367 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:48.710263    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9" event={"ID":"3925522f-d646-4bec-b1c1-58d3383380df","Type":"ContainerStarted","Data":"46568f16a907c687530685209b8609507d1a9ed43b9a3dafe338c339f545771a"}
Sep 17 15:40:48.710367 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:48.710336    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9" event={"ID":"3925522f-d646-4bec-b1c1-58d3383380df","Type":"ContainerStarted","Data":"ded596028ea2c7b10f42283f728179d4c885533d50a8b2709e0594fe6a1b0561"}
Sep 17 15:40:48.722407 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:48.721888    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5885cd8684-22xxn" podStartSLOduration=1.6556526809999998 podStartE2EDuration="5.721866714s" totalImagesPullingTime="4.066214033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:48.720670194 +0000 UTC m=+114.192510041" watchObservedRunningTime="2026-09-17 15:40:48.721866714 +0000 UTC m=+114.193706561"
Sep 17 15:40:48.741930 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:48.741815    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7f9446879-llkrn" podStartSLOduration=1.69257216 podStartE2EDuration="4.741795955s" totalImagesPullingTime="3.049223795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:48.740836063 +0000 UTC m=+114.212675910" watchObservedRunningTime="2026-09-17 15:40:48.741795955 +0000 UTC m=+114.213635801"
Sep 17 15:40:49.557459 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:49.557416    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:49.665001 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:49.664963    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:49.716750 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:49.716710    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9" event={"ID":"3925522f-d646-4bec-b1c1-58d3383380df","Type":"ContainerStarted","Data":"e4bbdd9dfc2dfc3164f8ab02f85ec4ec9d72d1a94163bfe5d2bf46ce2ffb0a19"}
Sep 17 15:40:49.717204 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:49.716837    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:49.720055 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:49.720030    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerStarted","Data":"399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41"}
Sep 17 15:40:49.720156 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:49.720087    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerStarted","Data":"de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace"}
Sep 17 15:40:49.720156 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:49.720119    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerStarted","Data":"5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc"}
Sep 17 15:40:49.720156 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:49.720146    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerStarted","Data":"735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731"}
Sep 17 15:40:49.738996 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:49.738958    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9" podStartSLOduration=2.5301151060000002 podStartE2EDuration="7.738945459s" totalImagesPullingTime="5.208830353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:49.738237438 +0000 UTC m=+115.210077288" watchObservedRunningTime="2026-09-17 15:40:49.738945459 +0000 UTC m=+115.210785288"
Sep 17 15:40:50.730123 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:50.730098    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7948667c77-xgmc9"
Sep 17 15:40:50.753939 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:50.753887    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.09427018 podStartE2EDuration="9.753870752s" totalImagesPullingTime="5.630529086s" totalInitContainerRuntime="29.071486ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:49.766888066 +0000 UTC m=+115.238727912" watchObservedRunningTime="2026-09-17 15:40:50.753870752 +0000 UTC m=+116.225710598"
Sep 17 15:40:53.541407 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:53.541354    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:53.642625 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:53.642585    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:53.807089 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:53.806996    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:53.874213 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:53.874186    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:54.120817 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:54.120789    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:54.121021 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:54.120852    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:54.125696 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:54.125673    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:54.745506 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:54.745479    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:40:55.311230 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:55.311189    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:55.386766 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:55.386729    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:55.775828 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:55.775798    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:55.837212 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:55.837180    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:56.159970 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:56.159931    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:56.215715 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:56.215682    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:57.396027 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:57.395970    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:57.497178 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:57.497134    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:58.004574 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:58.004535    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:58.106574 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:40:58.106540    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:01.806240 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:01.806210    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:01.907769 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:01.907725    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:02.658350 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:02.658308    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:02.759079 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:02.759044    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:03.213424 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:03.213389    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:03.278824 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:03.278784    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:04.779404 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:04.779319    2574 generic.go:356] "Generic (PLEG): container finished" podID="57985d52-f04f-4487-84de-ddfd56e490af" containerID="b0fdce84f2cc3a3a93142b57b5264e98d985b87fd083b1b783c4eaacdf1e2e41" exitCode=0
Sep 17 15:41:04.779404 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:04.779369    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk" event={"ID":"57985d52-f04f-4487-84de-ddfd56e490af","Type":"ContainerDied","Data":"b0fdce84f2cc3a3a93142b57b5264e98d985b87fd083b1b783c4eaacdf1e2e41"}
Sep 17 15:41:04.779857 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:04.779675    2574 scope.go:117] "RemoveContainer" containerID="b0fdce84f2cc3a3a93142b57b5264e98d985b87fd083b1b783c4eaacdf1e2e41"
Sep 17 15:41:04.851195 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:04.851173    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs\") pod \"network-metrics-daemon-c4llb\" (UID: \"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf\") " pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:41:04.853778 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:04.853756    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf-metrics-certs\") pod \"network-metrics-daemon-c4llb\" (UID: \"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf\") " pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:41:04.867064 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:04.867035    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:41:04.867212 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:04.867083    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:41:04.961182 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:04.961157    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j5497\""
Sep 17 15:41:04.969190 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:04.969170    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c4llb"
Sep 17 15:41:05.243084 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:05.243046    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c4llb"]
Sep 17 15:41:05.245797 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:41:05.245769    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode751638c_7ed5_4d66_ba1d_1f9a9cb9eeaf.slice/crio-2fbfe966a7059918be4c4f272df8e60e1e88d7939e61064ecc0e5821ea81552e WatchSource:0}: Error finding container 2fbfe966a7059918be4c4f272df8e60e1e88d7939e61064ecc0e5821ea81552e: Status 404 returned error can't find the container with id 2fbfe966a7059918be4c4f272df8e60e1e88d7939e61064ecc0e5821ea81552e
Sep 17 15:41:05.573409 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:05.573340    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5885cd8684-22xxn"]
Sep 17 15:41:05.785887 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:05.785847    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-mf6mk" event={"ID":"57985d52-f04f-4487-84de-ddfd56e490af","Type":"ContainerStarted","Data":"dba0e7853c00f64bbf83afeafee0d2f0e8f2cdbaeaeedb04ea8b7149c1cd6d02"}
Sep 17 15:41:05.787195 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:05.787149    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4llb" event={"ID":"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf","Type":"ContainerStarted","Data":"2fbfe966a7059918be4c4f272df8e60e1e88d7939e61064ecc0e5821ea81552e"}
Sep 17 15:41:06.792484 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:06.792416    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4llb" event={"ID":"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf","Type":"ContainerStarted","Data":"6c57e5f1abe06d149b044b8dd16666c1a5985d58ccc34d28f08243a1b30928ab"}
Sep 17 15:41:06.792901 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:06.792505    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c4llb" event={"ID":"e751638c-7ed5-4d66-ba1d-1f9a9cb9eeaf","Type":"ContainerStarted","Data":"3ba95d55a36fd56002a67f741b4ddae0492436b427d72b3ba3eeba454cc0f690"}
Sep 17 15:41:06.809031 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:06.808975    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c4llb" podStartSLOduration=129.945469063 podStartE2EDuration="2m10.808957106s" totalImagesPullingTime="863.488043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:06.807378315 +0000 UTC m=+132.279218160" watchObservedRunningTime="2026-09-17 15:41:06.808957106 +0000 UTC m=+132.280796952"
Sep 17 15:41:06.903120 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:06.903080    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:06.960162 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:06.960128    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:07.208505 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:07.208433    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:07.268031 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:07.267996    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:07.802055 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:07.802015    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:07.904239 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:07.904206    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:09.174356 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:09.174322    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64b88459dd-xdnj5_1d1c6522-62ea-4c17-b992-33f236073899/router/0.log"
Sep 17 15:41:09.195338 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:09.195316    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lpgls_02f7fe3e-05c4-49ac-97d2-35337336c9cb/serve-healthcheck-canary/0.log"
Sep 17 15:41:09.506231 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:09.506141    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:09.607157 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:09.607117    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:09.662747 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:09.662715    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:09.927203 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:09.927167    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:16.385403 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:16.385365    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:16.486068 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:16.486036    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:24.872189 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:24.872158    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:41:24.875827 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:24.875798    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7f9446879-llkrn"
Sep 17 15:41:30.594302 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.594262    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5885cd8684-22xxn" podUID="eb0eb4cb-25c7-4040-98f4-677d4677598c" containerName="console" containerID="cri-o://7798773414eebf8a837e80dca98e0d8f386e8f7697cf6bc8a3c7217b9a1772a5" gracePeriod=15
Sep 17 15:41:30.826881 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.826853    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:41:30.905923 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.905885    2574 generic.go:356] "Generic (PLEG): container finished" podID="eb0eb4cb-25c7-4040-98f4-677d4677598c" containerID="7798773414eebf8a837e80dca98e0d8f386e8f7697cf6bc8a3c7217b9a1772a5" exitCode=0
Sep 17 15:41:30.906076 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.905924    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5885cd8684-22xxn" event={"ID":"eb0eb4cb-25c7-4040-98f4-677d4677598c","Type":"ContainerDied","Data":"7798773414eebf8a837e80dca98e0d8f386e8f7697cf6bc8a3c7217b9a1772a5"}
Sep 17 15:41:30.906076 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.905979    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5885cd8684-22xxn" event={"ID":"eb0eb4cb-25c7-4040-98f4-677d4677598c","Type":"ContainerDied","Data":"e6610502443a8c99a461842c08e042def3cd94e10851c1ffa5aaa892c970d81f"}
Sep 17 15:41:30.906076 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.906000    2574 scope.go:117] "RemoveContainer" containerID="7798773414eebf8a837e80dca98e0d8f386e8f7697cf6bc8a3c7217b9a1772a5"
Sep 17 15:41:30.906076 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.905935    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5885cd8684-22xxn"
Sep 17 15:41:30.913936 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.913920    2574 scope.go:117] "RemoveContainer" containerID="7798773414eebf8a837e80dca98e0d8f386e8f7697cf6bc8a3c7217b9a1772a5"
Sep 17 15:41:30.914218 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:41:30.914198    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7798773414eebf8a837e80dca98e0d8f386e8f7697cf6bc8a3c7217b9a1772a5\": container with ID starting with 7798773414eebf8a837e80dca98e0d8f386e8f7697cf6bc8a3c7217b9a1772a5 not found: ID does not exist" containerID="7798773414eebf8a837e80dca98e0d8f386e8f7697cf6bc8a3c7217b9a1772a5"
Sep 17 15:41:30.914276 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.914228    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7798773414eebf8a837e80dca98e0d8f386e8f7697cf6bc8a3c7217b9a1772a5"} err="failed to get container status \"7798773414eebf8a837e80dca98e0d8f386e8f7697cf6bc8a3c7217b9a1772a5\": rpc error: code = NotFound desc = could not find container \"7798773414eebf8a837e80dca98e0d8f386e8f7697cf6bc8a3c7217b9a1772a5\": container with ID starting with 7798773414eebf8a837e80dca98e0d8f386e8f7697cf6bc8a3c7217b9a1772a5 not found: ID does not exist"
Sep 17 15:41:30.972578 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.972544    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-service-ca\") pod \"eb0eb4cb-25c7-4040-98f4-677d4677598c\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") "
Sep 17 15:41:30.972578 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.972583    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-oauth-config\") pod \"eb0eb4cb-25c7-4040-98f4-677d4677598c\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") "
Sep 17 15:41:30.972743 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.972609    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-286x2\" (UniqueName: \"kubernetes.io/projected/eb0eb4cb-25c7-4040-98f4-677d4677598c-kube-api-access-286x2\") pod \"eb0eb4cb-25c7-4040-98f4-677d4677598c\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") "
Sep 17 15:41:30.972743 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.972633    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-serving-cert\") pod \"eb0eb4cb-25c7-4040-98f4-677d4677598c\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") "
Sep 17 15:41:30.972743 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.972718    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-oauth-serving-cert\") pod \"eb0eb4cb-25c7-4040-98f4-677d4677598c\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") "
Sep 17 15:41:30.972875 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.972845    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-config\") pod \"eb0eb4cb-25c7-4040-98f4-677d4677598c\" (UID: \"eb0eb4cb-25c7-4040-98f4-677d4677598c\") "
Sep 17 15:41:30.973039 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.973011    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-service-ca" (OuterVolumeSpecName: "service-ca") pod "eb0eb4cb-25c7-4040-98f4-677d4677598c" (UID: "eb0eb4cb-25c7-4040-98f4-677d4677598c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:41:30.973129 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.973083    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eb0eb4cb-25c7-4040-98f4-677d4677598c" (UID: "eb0eb4cb-25c7-4040-98f4-677d4677598c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:41:30.973187 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.973175    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:41:30.973242 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.973190    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-oauth-serving-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:41:30.973242 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.973218    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-config" (OuterVolumeSpecName: "console-config") pod "eb0eb4cb-25c7-4040-98f4-677d4677598c" (UID: "eb0eb4cb-25c7-4040-98f4-677d4677598c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:41:30.974957 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.974938    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eb0eb4cb-25c7-4040-98f4-677d4677598c" (UID: "eb0eb4cb-25c7-4040-98f4-677d4677598c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:41:30.975022 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.974959    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eb0eb4cb-25c7-4040-98f4-677d4677598c" (UID: "eb0eb4cb-25c7-4040-98f4-677d4677598c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:41:30.975071 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:30.975052    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0eb4cb-25c7-4040-98f4-677d4677598c-kube-api-access-286x2" (OuterVolumeSpecName: "kube-api-access-286x2") pod "eb0eb4cb-25c7-4040-98f4-677d4677598c" (UID: "eb0eb4cb-25c7-4040-98f4-677d4677598c"). InnerVolumeSpecName "kube-api-access-286x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:41:31.074521 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:31.074494    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-config\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:41:31.074521 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:31.074519    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-oauth-config\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:41:31.074647 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:31.074530    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-286x2\" (UniqueName: \"kubernetes.io/projected/eb0eb4cb-25c7-4040-98f4-677d4677598c-kube-api-access-286x2\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:41:31.074647 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:31.074540    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb0eb4cb-25c7-4040-98f4-677d4677598c-console-serving-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:41:31.220954 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:31.220890    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5885cd8684-22xxn"]
Sep 17 15:41:31.224715 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:31.224695    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5885cd8684-22xxn"]
Sep 17 15:41:31.382964 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:31.382930    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:31.443186 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:31.443156    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:33.047870 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:33.047836    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb0eb4cb-25c7-4040-98f4-677d4677598c" path="/var/lib/kubelet/pods/eb0eb4cb-25c7-4040-98f4-677d4677598c/volumes"
Sep 17 15:41:59.386308 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:59.386272    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:59.811070 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:41:59.810983    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:00.456453 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:00.456402    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:01.018168 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:01.018130    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="alertmanager" containerID="cri-o://20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6" gracePeriod=120
Sep 17 15:42:01.018367 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:01.018227    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="kube-rbac-proxy" containerID="cri-o://5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc" gracePeriod=120
Sep 17 15:42:01.018367 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:01.018267    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="prom-label-proxy" containerID="cri-o://399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41" gracePeriod=120
Sep 17 15:42:01.018367 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:01.018215    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="kube-rbac-proxy-metric" containerID="cri-o://de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace" gracePeriod=120
Sep 17 15:42:01.018367 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:01.018337    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="config-reloader" containerID="cri-o://ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f" gracePeriod=120
Sep 17 15:42:01.018367 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:01.018226    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="kube-rbac-proxy-web" containerID="cri-o://735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731" gracePeriod=120
Sep 17 15:42:02.025105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.025075    2574 generic.go:356] "Generic (PLEG): container finished" podID="029afaa8-e959-441d-a9e0-6c80551fb465" containerID="399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41" exitCode=0
Sep 17 15:42:02.025105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.025098    2574 generic.go:356] "Generic (PLEG): container finished" podID="029afaa8-e959-441d-a9e0-6c80551fb465" containerID="5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc" exitCode=0
Sep 17 15:42:02.025105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.025104    2574 generic.go:356] "Generic (PLEG): container finished" podID="029afaa8-e959-441d-a9e0-6c80551fb465" containerID="ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f" exitCode=0
Sep 17 15:42:02.025105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.025110    2574 generic.go:356] "Generic (PLEG): container finished" podID="029afaa8-e959-441d-a9e0-6c80551fb465" containerID="20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6" exitCode=0
Sep 17 15:42:02.025569 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.025141    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerDied","Data":"399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41"}
Sep 17 15:42:02.025569 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.025204    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerDied","Data":"5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc"}
Sep 17 15:42:02.025569 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.025231    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerDied","Data":"ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f"}
Sep 17 15:42:02.025569 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.025255    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerDied","Data":"20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6"}
Sep 17 15:42:02.755182 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.755153    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:02.807739 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.807699    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/029afaa8-e959-441d-a9e0-6c80551fb465-alertmanager-main-db\") pod \"029afaa8-e959-441d-a9e0-6c80551fb465\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") "
Sep 17 15:42:02.807908 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.807742    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8njn\" (UniqueName: \"kubernetes.io/projected/029afaa8-e959-441d-a9e0-6c80551fb465-kube-api-access-m8njn\") pod \"029afaa8-e959-441d-a9e0-6c80551fb465\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") "
Sep 17 15:42:02.807908 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.807772    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-config-volume\") pod \"029afaa8-e959-441d-a9e0-6c80551fb465\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") "
Sep 17 15:42:02.807908 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.807806    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy\") pod \"029afaa8-e959-441d-a9e0-6c80551fb465\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") "
Sep 17 15:42:02.807908 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.807836    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/029afaa8-e959-441d-a9e0-6c80551fb465-config-out\") pod \"029afaa8-e959-441d-a9e0-6c80551fb465\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") "
Sep 17 15:42:02.807908 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.807868    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy-metric\") pod \"029afaa8-e959-441d-a9e0-6c80551fb465\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") "
Sep 17 15:42:02.808155 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.807909    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy-web\") pod \"029afaa8-e959-441d-a9e0-6c80551fb465\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") "
Sep 17 15:42:02.808155 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.807945    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-cluster-tls-config\") pod \"029afaa8-e959-441d-a9e0-6c80551fb465\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") "
Sep 17 15:42:02.808155 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.807980    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-web-config\") pod \"029afaa8-e959-441d-a9e0-6c80551fb465\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") "
Sep 17 15:42:02.808155 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.808013    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/029afaa8-e959-441d-a9e0-6c80551fb465-metrics-client-ca\") pod \"029afaa8-e959-441d-a9e0-6c80551fb465\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") "
Sep 17 15:42:02.808155 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.808039    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029afaa8-e959-441d-a9e0-6c80551fb465-alertmanager-trusted-ca-bundle\") pod \"029afaa8-e959-441d-a9e0-6c80551fb465\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") "
Sep 17 15:42:02.808155 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.808067    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-main-tls\") pod \"029afaa8-e959-441d-a9e0-6c80551fb465\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") "
Sep 17 15:42:02.808155 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.808074    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029afaa8-e959-441d-a9e0-6c80551fb465-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "029afaa8-e959-441d-a9e0-6c80551fb465" (UID: "029afaa8-e959-441d-a9e0-6c80551fb465"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:42:02.808155 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.808125    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/029afaa8-e959-441d-a9e0-6c80551fb465-tls-assets\") pod \"029afaa8-e959-441d-a9e0-6c80551fb465\" (UID: \"029afaa8-e959-441d-a9e0-6c80551fb465\") "
Sep 17 15:42:02.808569 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.808323    2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/029afaa8-e959-441d-a9e0-6c80551fb465-alertmanager-main-db\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.809212 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.808635    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029afaa8-e959-441d-a9e0-6c80551fb465-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "029afaa8-e959-441d-a9e0-6c80551fb465" (UID: "029afaa8-e959-441d-a9e0-6c80551fb465"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:02.809212 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.808959    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029afaa8-e959-441d-a9e0-6c80551fb465-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "029afaa8-e959-441d-a9e0-6c80551fb465" (UID: "029afaa8-e959-441d-a9e0-6c80551fb465"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:02.811954 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.811821    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "029afaa8-e959-441d-a9e0-6c80551fb465" (UID: "029afaa8-e959-441d-a9e0-6c80551fb465"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:02.811954 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.811928    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-config-volume" (OuterVolumeSpecName: "config-volume") pod "029afaa8-e959-441d-a9e0-6c80551fb465" (UID: "029afaa8-e959-441d-a9e0-6c80551fb465"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:02.812115 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.811950    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029afaa8-e959-441d-a9e0-6c80551fb465-kube-api-access-m8njn" (OuterVolumeSpecName: "kube-api-access-m8njn") pod "029afaa8-e959-441d-a9e0-6c80551fb465" (UID: "029afaa8-e959-441d-a9e0-6c80551fb465"). InnerVolumeSpecName "kube-api-access-m8njn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:42:02.812180 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.812146    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "029afaa8-e959-441d-a9e0-6c80551fb465" (UID: "029afaa8-e959-441d-a9e0-6c80551fb465"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:02.812418 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.812393    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "029afaa8-e959-441d-a9e0-6c80551fb465" (UID: "029afaa8-e959-441d-a9e0-6c80551fb465"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:02.812705 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.812667    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029afaa8-e959-441d-a9e0-6c80551fb465-config-out" (OuterVolumeSpecName: "config-out") pod "029afaa8-e959-441d-a9e0-6c80551fb465" (UID: "029afaa8-e959-441d-a9e0-6c80551fb465"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:42:02.812705 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.812678    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029afaa8-e959-441d-a9e0-6c80551fb465-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "029afaa8-e959-441d-a9e0-6c80551fb465" (UID: "029afaa8-e959-441d-a9e0-6c80551fb465"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:42:02.813300 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.813278    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "029afaa8-e959-441d-a9e0-6c80551fb465" (UID: "029afaa8-e959-441d-a9e0-6c80551fb465"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:02.816128 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.816002    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "029afaa8-e959-441d-a9e0-6c80551fb465" (UID: "029afaa8-e959-441d-a9e0-6c80551fb465"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:02.822569 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.822548    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-web-config" (OuterVolumeSpecName: "web-config") pod "029afaa8-e959-441d-a9e0-6c80551fb465" (UID: "029afaa8-e959-441d-a9e0-6c80551fb465"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:02.909034 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.908947    2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/029afaa8-e959-441d-a9e0-6c80551fb465-tls-assets\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.909034 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.908978    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8njn\" (UniqueName: \"kubernetes.io/projected/029afaa8-e959-441d-a9e0-6c80551fb465-kube-api-access-m8njn\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.909034 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.908989    2574 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.909034 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.908998    2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.909034 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.909007    2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/029afaa8-e959-441d-a9e0-6c80551fb465-config-out\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.909034 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.909016    2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.909034 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.909026    2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.909034 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.909034    2574 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-cluster-tls-config\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.909034 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.909043    2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-web-config\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.909404 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.909052    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/029afaa8-e959-441d-a9e0-6c80551fb465-metrics-client-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.909404 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.909060    2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/029afaa8-e959-441d-a9e0-6c80551fb465-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.909404 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:02.909068    2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/029afaa8-e959-441d-a9e0-6c80551fb465-secret-alertmanager-main-tls\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:42:03.033796 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.033760    2574 generic.go:356] "Generic (PLEG): container finished" podID="029afaa8-e959-441d-a9e0-6c80551fb465" containerID="de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace" exitCode=0
Sep 17 15:42:03.033796 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.033788    2574 generic.go:356] "Generic (PLEG): container finished" podID="029afaa8-e959-441d-a9e0-6c80551fb465" containerID="735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731" exitCode=0
Sep 17 15:42:03.034248 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.033825    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.034248 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.033847    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerDied","Data":"de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace"}
Sep 17 15:42:03.034248 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.033904    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerDied","Data":"735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731"}
Sep 17 15:42:03.034248 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.033931    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"029afaa8-e959-441d-a9e0-6c80551fb465","Type":"ContainerDied","Data":"a1f920d63d373a1ff4c9a1fe27f342ddef2208b3677d1113de9a22a8afe80a3d"}
Sep 17 15:42:03.034248 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.033962    2574 scope.go:117] "RemoveContainer" containerID="399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41"
Sep 17 15:42:03.041504 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.041479    2574 scope.go:117] "RemoveContainer" containerID="de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace"
Sep 17 15:42:03.048604 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.048584    2574 scope.go:117] "RemoveContainer" containerID="5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc"
Sep 17 15:42:03.055029 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.055008    2574 scope.go:117] "RemoveContainer" containerID="735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731"
Sep 17 15:42:03.060266 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.059667    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:03.062042 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.062025    2574 scope.go:117] "RemoveContainer" containerID="ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f"
Sep 17 15:42:03.067572 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.067548    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:03.069117 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.069099    2574 scope.go:117] "RemoveContainer" containerID="20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6"
Sep 17 15:42:03.075754 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.075737    2574 scope.go:117] "RemoveContainer" containerID="67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063"
Sep 17 15:42:03.082313 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.082295    2574 scope.go:117] "RemoveContainer" containerID="399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41"
Sep 17 15:42:03.082578 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:42:03.082560    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41\": container with ID starting with 399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41 not found: ID does not exist" containerID="399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41"
Sep 17 15:42:03.082635 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.082586    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41"} err="failed to get container status \"399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41\": rpc error: code = NotFound desc = could not find container \"399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41\": container with ID starting with 399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41 not found: ID does not exist"
Sep 17 15:42:03.082635 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.082607    2574 scope.go:117] "RemoveContainer" containerID="de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace"
Sep 17 15:42:03.082859 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:42:03.082843    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace\": container with ID starting with de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace not found: ID does not exist" containerID="de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace"
Sep 17 15:42:03.082902 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.082869    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace"} err="failed to get container status \"de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace\": rpc error: code = NotFound desc = could not find container \"de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace\": container with ID starting with de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace not found: ID does not exist"
Sep 17 15:42:03.082902 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.082885    2574 scope.go:117] "RemoveContainer" containerID="5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc"
Sep 17 15:42:03.083123 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:42:03.083107    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc\": container with ID starting with 5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc not found: ID does not exist" containerID="5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc"
Sep 17 15:42:03.083167 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.083129    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc"} err="failed to get container status \"5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc\": rpc error: code = NotFound desc = could not find container \"5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc\": container with ID starting with 5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc not found: ID does not exist"
Sep 17 15:42:03.083167 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.083143    2574 scope.go:117] "RemoveContainer" containerID="735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731"
Sep 17 15:42:03.083363 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:42:03.083349    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731\": container with ID starting with 735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731 not found: ID does not exist" containerID="735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731"
Sep 17 15:42:03.083398 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.083366    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731"} err="failed to get container status \"735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731\": rpc error: code = NotFound desc = could not find container \"735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731\": container with ID starting with 735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731 not found: ID does not exist"
Sep 17 15:42:03.083398 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.083377    2574 scope.go:117] "RemoveContainer" containerID="ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f"
Sep 17 15:42:03.083629 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:42:03.083613    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f\": container with ID starting with ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f not found: ID does not exist" containerID="ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f"
Sep 17 15:42:03.083667 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.083634    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f"} err="failed to get container status \"ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f\": rpc error: code = NotFound desc = could not find container \"ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f\": container with ID starting with ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f not found: ID does not exist"
Sep 17 15:42:03.083667 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.083650    2574 scope.go:117] "RemoveContainer" containerID="20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6"
Sep 17 15:42:03.083862 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:42:03.083843    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6\": container with ID starting with 20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6 not found: ID does not exist" containerID="20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6"
Sep 17 15:42:03.083900 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.083865    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6"} err="failed to get container status \"20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6\": rpc error: code = NotFound desc = could not find container \"20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6\": container with ID starting with 20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6 not found: ID does not exist"
Sep 17 15:42:03.083900 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.083879    2574 scope.go:117] "RemoveContainer" containerID="67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063"
Sep 17 15:42:03.084077 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:42:03.084064    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063\": container with ID starting with 67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063 not found: ID does not exist" containerID="67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063"
Sep 17 15:42:03.084111 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.084079    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063"} err="failed to get container status \"67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063\": rpc error: code = NotFound desc = could not find container \"67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063\": container with ID starting with 67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063 not found: ID does not exist"
Sep 17 15:42:03.084111 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.084091    2574 scope.go:117] "RemoveContainer" containerID="399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41"
Sep 17 15:42:03.084307 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.084291    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41"} err="failed to get container status \"399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41\": rpc error: code = NotFound desc = could not find container \"399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41\": container with ID starting with 399e2c8ed30cbcb7182d8595a2540927723b71f6e80bb1a6c642e67b79644c41 not found: ID does not exist"
Sep 17 15:42:03.084345 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.084308    2574 scope.go:117] "RemoveContainer" containerID="de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace"
Sep 17 15:42:03.084525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.084510    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace"} err="failed to get container status \"de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace\": rpc error: code = NotFound desc = could not find container \"de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace\": container with ID starting with de2abe7e2c71cbb48c5c645a9bbb0cb252ea5558b20ed169f4e3145f98b84ace not found: ID does not exist"
Sep 17 15:42:03.084565 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.084525    2574 scope.go:117] "RemoveContainer" containerID="5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc"
Sep 17 15:42:03.084736 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.084719    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc"} err="failed to get container status \"5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc\": rpc error: code = NotFound desc = could not find container \"5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc\": container with ID starting with 5fb75153dd2e5b28f6be12a94ec1bee1ec0bff2e3a3c076cc1ab484552084ecc not found: ID does not exist"
Sep 17 15:42:03.084772 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.084737    2574 scope.go:117] "RemoveContainer" containerID="735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731"
Sep 17 15:42:03.084937 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.084923    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731"} err="failed to get container status \"735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731\": rpc error: code = NotFound desc = could not find container \"735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731\": container with ID starting with 735a0822de3cf0ff4a6b8392f94740198b7286b31a806ab9ca01520a38347731 not found: ID does not exist"
Sep 17 15:42:03.084972 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.084937    2574 scope.go:117] "RemoveContainer" containerID="ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f"
Sep 17 15:42:03.085158 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.085144    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f"} err="failed to get container status \"ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f\": rpc error: code = NotFound desc = could not find container \"ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f\": container with ID starting with ca7f9f4038ab0ef35958dd727cf6497d3afd8323243e643fab51704c2e31569f not found: ID does not exist"
Sep 17 15:42:03.085194 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.085159    2574 scope.go:117] "RemoveContainer" containerID="20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6"
Sep 17 15:42:03.085375 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.085357    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6"} err="failed to get container status \"20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6\": rpc error: code = NotFound desc = could not find container \"20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6\": container with ID starting with 20343201b64e8f2326fe9d625da14805d843b65aebee17b8f488ceeabdc9fae6 not found: ID does not exist"
Sep 17 15:42:03.085421 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.085377    2574 scope.go:117] "RemoveContainer" containerID="67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063"
Sep 17 15:42:03.085636 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.085620    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063"} err="failed to get container status \"67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063\": rpc error: code = NotFound desc = could not find container \"67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063\": container with ID starting with 67cb450de4f956d5b8e365c2782f257466637b6df0e8ecf1c4f0e3f928e86063 not found: ID does not exist"
Sep 17 15:42:03.100352 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100323    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:03.100622 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100609    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb0eb4cb-25c7-4040-98f4-677d4677598c" containerName="console"
Sep 17 15:42:03.100667 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100624    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb0eb4cb-25c7-4040-98f4-677d4677598c" containerName="console"
Sep 17 15:42:03.100667 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100635    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="kube-rbac-proxy-metric"
Sep 17 15:42:03.100667 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100641    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="kube-rbac-proxy-metric"
Sep 17 15:42:03.100667 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100653    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="kube-rbac-proxy"
Sep 17 15:42:03.100667 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100659    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="kube-rbac-proxy"
Sep 17 15:42:03.100667 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100665    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="prom-label-proxy"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100671    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="prom-label-proxy"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100680    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="alertmanager"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100685    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="alertmanager"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100691    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="kube-rbac-proxy-web"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100696    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="kube-rbac-proxy-web"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100705    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="init-config-reloader"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100710    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="init-config-reloader"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100716    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="config-reloader"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100721    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="config-reloader"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100760    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="prom-label-proxy"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100767    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb0eb4cb-25c7-4040-98f4-677d4677598c" containerName="console"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100773    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="kube-rbac-proxy"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100779    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="kube-rbac-proxy-metric"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100784    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="kube-rbac-proxy-web"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100791    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="config-reloader"
Sep 17 15:42:03.100829 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.100798    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" containerName="alertmanager"
Sep 17 15:42:03.106003 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.105919    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.109161 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.108775    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 15:42:03.109161 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.109066    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 15:42:03.109161 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.109081    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 15:42:03.109161 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.109086    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 15:42:03.109161 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.109082    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 15:42:03.109161 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.109136    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9lrw6\""
Sep 17 15:42:03.109648 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.109506    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 15:42:03.109901 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.109883    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 15:42:03.109955 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.109928    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 15:42:03.116141 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.116108    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 15:42:03.121303 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.121276    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:03.210852 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.210754    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.210852 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.210798    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.210852 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.210822    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71c12437-d48a-463a-a015-337c239986ac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.211113 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.210898    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.211113 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.210948    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.211113 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.211005    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2cxv\" (UniqueName: \"kubernetes.io/projected/71c12437-d48a-463a-a015-337c239986ac-kube-api-access-k2cxv\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.211113 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.211023    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71c12437-d48a-463a-a015-337c239986ac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.211113 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.211041    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/71c12437-d48a-463a-a015-337c239986ac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.211113 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.211056    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.211113 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.211087    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-web-config\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.211390 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.211117    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71c12437-d48a-463a-a015-337c239986ac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.211390 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.211184    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-config-volume\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.211390 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.211208    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71c12437-d48a-463a-a015-337c239986ac-config-out\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.312231 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.312198    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.312321 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.312238    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.312321 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.312262    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71c12437-d48a-463a-a015-337c239986ac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.312321 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.312284    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.312479 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.312412    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.312517 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.312494    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2cxv\" (UniqueName: \"kubernetes.io/projected/71c12437-d48a-463a-a015-337c239986ac-kube-api-access-k2cxv\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.312560 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.312516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71c12437-d48a-463a-a015-337c239986ac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.312560 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.312535    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/71c12437-d48a-463a-a015-337c239986ac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.312641 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.312557    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.312641 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.312584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-web-config\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.312641 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.312605    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71c12437-d48a-463a-a015-337c239986ac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.312794 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.312654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-config-volume\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.312794 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.312675    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71c12437-d48a-463a-a015-337c239986ac-config-out\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.313281 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.313254    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71c12437-d48a-463a-a015-337c239986ac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.314388 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.314078    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71c12437-d48a-463a-a015-337c239986ac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.315525 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.315495    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.315622 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.315495    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.315740 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.315714    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.315805 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.315789    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.315860 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.315801    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71c12437-d48a-463a-a015-337c239986ac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.315860 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.315830    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-web-config\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.315928 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.315872    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/71c12437-d48a-463a-a015-337c239986ac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.316096 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.316081    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.316790 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.316771    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/71c12437-d48a-463a-a015-337c239986ac-config-volume\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.317749 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.317733    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71c12437-d48a-463a-a015-337c239986ac-config-out\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.320975 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.320953    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2cxv\" (UniqueName: \"kubernetes.io/projected/71c12437-d48a-463a-a015-337c239986ac-kube-api-access-k2cxv\") pod \"alertmanager-main-0\" (UID: \"71c12437-d48a-463a-a015-337c239986ac\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.422613 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.422579    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.701501 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:03.701456    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:03.705327 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:42:03.705299    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c12437_d48a_463a_a015_337c239986ac.slice/crio-b2dd9e2ce3f4ac28a3ada096efcb97bef0152496cf7f13a571af7315d56f1263 WatchSource:0}: Error finding container b2dd9e2ce3f4ac28a3ada096efcb97bef0152496cf7f13a571af7315d56f1263: Status 404 returned error can't find the container with id b2dd9e2ce3f4ac28a3ada096efcb97bef0152496cf7f13a571af7315d56f1263
Sep 17 15:42:04.040047 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.039960    2574 generic.go:356] "Generic (PLEG): container finished" podID="71c12437-d48a-463a-a015-337c239986ac" containerID="22f12103daf5d5786a10900fa298d0c13f0094ff83b20487b645ed5bac306bf9" exitCode=0
Sep 17 15:42:04.040047 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.040030    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71c12437-d48a-463a-a015-337c239986ac","Type":"ContainerDied","Data":"22f12103daf5d5786a10900fa298d0c13f0094ff83b20487b645ed5bac306bf9"}
Sep 17 15:42:04.040433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.040082    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71c12437-d48a-463a-a015-337c239986ac","Type":"ContainerStarted","Data":"b2dd9e2ce3f4ac28a3ada096efcb97bef0152496cf7f13a571af7315d56f1263"}
Sep 17 15:42:04.616962 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.616924    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"]
Sep 17 15:42:04.620203 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.620186    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.625192 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.625170    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 15:42:04.625395 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.625375    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 15:42:04.625522 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.625418    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 15:42:04.626019 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.625999    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 15:42:04.629768 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.629749    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 15:42:04.629918 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.629903    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-gzkmq\""
Sep 17 15:42:04.640480 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.640243    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 15:42:04.640971 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.640944    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"]
Sep 17 15:42:04.726457 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.726398    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6f79929-5d0a-4e50-898d-aac097380652-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.726457 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.726458    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6f79929-5d0a-4e50-898d-aac097380652-serving-certs-ca-bundle\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.726672 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.726536    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6f79929-5d0a-4e50-898d-aac097380652-metrics-client-ca\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.726672 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.726592    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6f79929-5d0a-4e50-898d-aac097380652-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.726672 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.726638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w42sr\" (UniqueName: \"kubernetes.io/projected/b6f79929-5d0a-4e50-898d-aac097380652-kube-api-access-w42sr\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.726816 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.726674    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b6f79929-5d0a-4e50-898d-aac097380652-secret-telemeter-client\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.726816 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.726717    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b6f79929-5d0a-4e50-898d-aac097380652-federate-client-tls\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.726816 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.726759    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b6f79929-5d0a-4e50-898d-aac097380652-telemeter-client-tls\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.827932 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.827889    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b6f79929-5d0a-4e50-898d-aac097380652-federate-client-tls\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.827932 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.827937    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b6f79929-5d0a-4e50-898d-aac097380652-telemeter-client-tls\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.828163 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.827968    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6f79929-5d0a-4e50-898d-aac097380652-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.828163 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.827990    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6f79929-5d0a-4e50-898d-aac097380652-serving-certs-ca-bundle\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.828163 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.828016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6f79929-5d0a-4e50-898d-aac097380652-metrics-client-ca\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.828163 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.828033    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6f79929-5d0a-4e50-898d-aac097380652-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.828163 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.828056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w42sr\" (UniqueName: \"kubernetes.io/projected/b6f79929-5d0a-4e50-898d-aac097380652-kube-api-access-w42sr\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.828163 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.828086    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b6f79929-5d0a-4e50-898d-aac097380652-secret-telemeter-client\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.828922 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.828887    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6f79929-5d0a-4e50-898d-aac097380652-serving-certs-ca-bundle\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.829127 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.828963    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6f79929-5d0a-4e50-898d-aac097380652-metrics-client-ca\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.829240 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.829149    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6f79929-5d0a-4e50-898d-aac097380652-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.831000 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.830974    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b6f79929-5d0a-4e50-898d-aac097380652-telemeter-client-tls\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.831135 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.831113    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6f79929-5d0a-4e50-898d-aac097380652-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.831174 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.831113    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b6f79929-5d0a-4e50-898d-aac097380652-federate-client-tls\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.831174 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.831126    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b6f79929-5d0a-4e50-898d-aac097380652-secret-telemeter-client\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.839390 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.839368    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w42sr\" (UniqueName: \"kubernetes.io/projected/b6f79929-5d0a-4e50-898d-aac097380652-kube-api-access-w42sr\") pod \"telemeter-client-c7dcb6fd8-s2rht\" (UID: \"b6f79929-5d0a-4e50-898d-aac097380652\") " pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:04.929125 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:04.929021    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"
Sep 17 15:42:05.048415 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:05.048382    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="029afaa8-e959-441d-a9e0-6c80551fb465" path="/var/lib/kubelet/pods/029afaa8-e959-441d-a9e0-6c80551fb465/volumes"
Sep 17 15:42:05.048937 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:05.048919    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71c12437-d48a-463a-a015-337c239986ac","Type":"ContainerStarted","Data":"b627dab8055b52e080fd45ad7188a81ca3946b5fe73eb536b499533033b3d16c"}
Sep 17 15:42:05.048982 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:05.048975    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71c12437-d48a-463a-a015-337c239986ac","Type":"ContainerStarted","Data":"3af445dfea3503346b172391e363cdd99c257eade8441cf8723a4a28bb038304"}
Sep 17 15:42:05.049017 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:05.049002    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71c12437-d48a-463a-a015-337c239986ac","Type":"ContainerStarted","Data":"38faca1f6294fc82f3849b93250bd1888a4563fbaee94b8d9ef1ff8bcc71f348"}
Sep 17 15:42:05.049049 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:05.049027    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71c12437-d48a-463a-a015-337c239986ac","Type":"ContainerStarted","Data":"eb8234cecaf2d73791d2168b5b13acd5db80fee281ac4c18466a8e3a51713551"}
Sep 17 15:42:05.049084 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:05.049054    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71c12437-d48a-463a-a015-337c239986ac","Type":"ContainerStarted","Data":"07f5afafe3ad9767f6ca367f6863ff3fc93f4fce54b8a8389ce24d534d8b9f52"}
Sep 17 15:42:05.049084 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:05.049078    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71c12437-d48a-463a-a015-337c239986ac","Type":"ContainerStarted","Data":"5ed0a1fced8345019e2d235a138290e0910c97b925ec5a5a92fd14ec9681db2a"}
Sep 17 15:42:05.075320 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:05.075262    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.055987321 podStartE2EDuration="2.07524683s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.259509ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:05.07478558 +0000 UTC m=+190.546625427" watchObservedRunningTime="2026-09-17 15:42:05.07524683 +0000 UTC m=+190.547086677"
Sep 17 15:42:05.213408 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:05.213379    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht"]
Sep 17 15:42:06.053824 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:06.053778    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht" event={"ID":"b6f79929-5d0a-4e50-898d-aac097380652","Type":"ContainerStarted","Data":"df7539e8ce89c1f17a83104fce4df083ba315697e6f6384fbf11ab784151c041"}
Sep 17 15:42:08.068055 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:08.068018    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht" event={"ID":"b6f79929-5d0a-4e50-898d-aac097380652","Type":"ContainerStarted","Data":"75d99928f1b6eec1174c34736aaae311aaa562383eeb842d33d4529604f7f293"}
Sep 17 15:42:08.068435 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:08.068100    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht" event={"ID":"b6f79929-5d0a-4e50-898d-aac097380652","Type":"ContainerStarted","Data":"fa057277967c22aa2620a2a54a8a5128f01822d317163dc7273f6954216c60ff"}
Sep 17 15:42:08.068435 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:08.068121    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht" event={"ID":"b6f79929-5d0a-4e50-898d-aac097380652","Type":"ContainerStarted","Data":"f06b923f49cf66a36c5aef4bdf1abc63cd1230496e148643fd0f6e53a6a51ebc"}
Sep 17 15:42:08.092300 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:08.092242    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-c7dcb6fd8-s2rht" podStartSLOduration=2.155091821 podStartE2EDuration="4.092225581s" totalImagesPullingTime="1.93713376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:08.091011366 +0000 UTC m=+193.562851222" watchObservedRunningTime="2026-09-17 15:42:08.092225581 +0000 UTC m=+193.564065459"
Sep 17 15:42:08.248175 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:08.248137    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:08.303817 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:08.303785    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:18.940854 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:18.940815    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:19.041141 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:19.041104    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:19.141933 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:19.141874    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:39.418799 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:42:39.418761    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:14.784042 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:14.784000    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:14.892749 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:14.892709    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:17.238957 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:17.238847    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:17.308459 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:17.308400    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:26.826189 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:26.826132    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:26.880014 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:26.879966    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:28.153694 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:28.153654    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:28.206680 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:28.206650    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:28.263132 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:28.263084    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:31.378236 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:31.378195    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:31.479521 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:31.479485    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:31.538649 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:31.538608    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:42.960790 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:42.960758    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:43.013579 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:43.013542    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:54.940514 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:54.940487    2574 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 15:43:59.673920 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:43:59.673883    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:44:03.535655 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:44:03.535606    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:00.769840 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:00.769758    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:03.642978 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:03.642941    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:23.161667 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:23.161631    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:23.346999 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:23.346955    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:28.462885 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:28.462829    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:28.753656 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:28.753553    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:28.833587 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:28.833514    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:29.008016 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.007891    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:29.222177 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.222139    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:29.300431 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.300348    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"]
Sep 17 15:45:29.303527 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.303510    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"
Sep 17 15:45:29.306493 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.306466    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 15:45:29.306639 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.306478    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 15:45:29.306639 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.306569    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 15:45:29.306639 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.306583    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 15:45:29.306639 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.306638    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 15:45:29.306867 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.306787    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-xhfnh\""
Sep 17 15:45:29.312913 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.312888    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"]
Sep 17 15:45:29.374192 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.374152    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhwpv\" (UniqueName: \"kubernetes.io/projected/7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc-kube-api-access-hhwpv\") pod \"keda-metrics-apiserver-77857b7b98-ksxg8\" (UID: \"7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"
Sep 17 15:45:29.374192 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.374200    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-ksxg8\" (UID: \"7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"
Sep 17 15:45:29.374417 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.374227    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc-certificates\") pod \"keda-metrics-apiserver-77857b7b98-ksxg8\" (UID: \"7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"
Sep 17 15:45:29.475319 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.475287    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-ksxg8\" (UID: \"7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"
Sep 17 15:45:29.475778 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.475332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc-certificates\") pod \"keda-metrics-apiserver-77857b7b98-ksxg8\" (UID: \"7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"
Sep 17 15:45:29.475778 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.475380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhwpv\" (UniqueName: \"kubernetes.io/projected/7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc-kube-api-access-hhwpv\") pod \"keda-metrics-apiserver-77857b7b98-ksxg8\" (UID: \"7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"
Sep 17 15:45:29.475778 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.475755    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-ksxg8\" (UID: \"7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"
Sep 17 15:45:29.478134 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.478110    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc-certificates\") pod \"keda-metrics-apiserver-77857b7b98-ksxg8\" (UID: \"7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"
Sep 17 15:45:29.484336 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.484313    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhwpv\" (UniqueName: \"kubernetes.io/projected/7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc-kube-api-access-hhwpv\") pod \"keda-metrics-apiserver-77857b7b98-ksxg8\" (UID: \"7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"
Sep 17 15:45:29.524370 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.524335    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:29.584380 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.584309    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-j5shx"]
Sep 17 15:45:29.587595 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.587577    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-j5shx"
Sep 17 15:45:29.589761 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.589739    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 15:45:29.598397 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.598372    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-j5shx"]
Sep 17 15:45:29.614591 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.614553    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"
Sep 17 15:45:29.619635 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.619596    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:29.677519 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.677475    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnl6z\" (UniqueName: \"kubernetes.io/projected/c96dbe53-72b1-4b8d-9bd3-274fead2201d-kube-api-access-gnl6z\") pod \"keda-admission-6c4d864b86-j5shx\" (UID: \"c96dbe53-72b1-4b8d-9bd3-274fead2201d\") " pod="openshift-keda/keda-admission-6c4d864b86-j5shx"
Sep 17 15:45:29.677661 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.677574    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c96dbe53-72b1-4b8d-9bd3-274fead2201d-certificates\") pod \"keda-admission-6c4d864b86-j5shx\" (UID: \"c96dbe53-72b1-4b8d-9bd3-274fead2201d\") " pod="openshift-keda/keda-admission-6c4d864b86-j5shx"
Sep 17 15:45:29.690388 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.690334    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:29.778036 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.777998    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnl6z\" (UniqueName: \"kubernetes.io/projected/c96dbe53-72b1-4b8d-9bd3-274fead2201d-kube-api-access-gnl6z\") pod \"keda-admission-6c4d864b86-j5shx\" (UID: \"c96dbe53-72b1-4b8d-9bd3-274fead2201d\") " pod="openshift-keda/keda-admission-6c4d864b86-j5shx"
Sep 17 15:45:29.778217 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.778066    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c96dbe53-72b1-4b8d-9bd3-274fead2201d-certificates\") pod \"keda-admission-6c4d864b86-j5shx\" (UID: \"c96dbe53-72b1-4b8d-9bd3-274fead2201d\") " pod="openshift-keda/keda-admission-6c4d864b86-j5shx"
Sep 17 15:45:29.781000 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.780967    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c96dbe53-72b1-4b8d-9bd3-274fead2201d-certificates\") pod \"keda-admission-6c4d864b86-j5shx\" (UID: \"c96dbe53-72b1-4b8d-9bd3-274fead2201d\") " pod="openshift-keda/keda-admission-6c4d864b86-j5shx"
Sep 17 15:45:29.787726 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.787381    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnl6z\" (UniqueName: \"kubernetes.io/projected/c96dbe53-72b1-4b8d-9bd3-274fead2201d-kube-api-access-gnl6z\") pod \"keda-admission-6c4d864b86-j5shx\" (UID: \"c96dbe53-72b1-4b8d-9bd3-274fead2201d\") " pod="openshift-keda/keda-admission-6c4d864b86-j5shx"
Sep 17 15:45:29.890038 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.890005    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:29.894480 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.894428    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"]
Sep 17 15:45:29.898458 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:45:29.898413    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa37baf_f8c4_4cf5_a1e3_ed020506a8dc.slice/crio-12613129bc2310c9684a3c6f101c4cc8d99edbd9c97e8da6dc05802672b74a84 WatchSource:0}: Error finding container 12613129bc2310c9684a3c6f101c4cc8d99edbd9c97e8da6dc05802672b74a84: Status 404 returned error can't find the container with id 12613129bc2310c9684a3c6f101c4cc8d99edbd9c97e8da6dc05802672b74a84
Sep 17 15:45:29.899934 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.899917    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:45:29.901570 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:29.901546    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-j5shx"
Sep 17 15:45:30.090641 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:30.090586    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:30.179018 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:30.178650    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-j5shx"]
Sep 17 15:45:30.181474 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:45:30.181429    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96dbe53_72b1_4b8d_9bd3_274fead2201d.slice/crio-f4c1f833bb24844a888abe60805e5b4838458cb0337bf4a7a542e6792a031f8a WatchSource:0}: Error finding container f4c1f833bb24844a888abe60805e5b4838458cb0337bf4a7a542e6792a031f8a: Status 404 returned error can't find the container with id f4c1f833bb24844a888abe60805e5b4838458cb0337bf4a7a542e6792a031f8a
Sep 17 15:45:30.182262 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:30.182228    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:30.812220 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:30.812133    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-j5shx" event={"ID":"c96dbe53-72b1-4b8d-9bd3-274fead2201d","Type":"ContainerStarted","Data":"f4c1f833bb24844a888abe60805e5b4838458cb0337bf4a7a542e6792a031f8a"}
Sep 17 15:45:30.814065 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:30.814010    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8" event={"ID":"7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc","Type":"ContainerStarted","Data":"12613129bc2310c9684a3c6f101c4cc8d99edbd9c97e8da6dc05802672b74a84"}
Sep 17 15:45:33.829383 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:33.829279    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-j5shx" event={"ID":"c96dbe53-72b1-4b8d-9bd3-274fead2201d","Type":"ContainerStarted","Data":"d0f36b305378eb3618f00e50fe02c25dca362cd6aa28cd9840c0347f4fe623ce"}
Sep 17 15:45:33.829383 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:33.829341    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-j5shx"
Sep 17 15:45:33.830915 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:33.830890    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8" event={"ID":"7fa37baf-f8c4-4cf5-a1e3-ed020506a8dc","Type":"ContainerStarted","Data":"f4e28de4bf23ab7faf927368ee881d42fd5d20852bf57860971f2d9b9f9b765c"}
Sep 17 15:45:33.831034 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:33.831008    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"
Sep 17 15:45:33.853479 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:33.850910    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-j5shx" podStartSLOduration=1.678983761 podStartE2EDuration="4.850884046s" totalImagesPullingTime="3.171900285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:33.847876967 +0000 UTC m=+399.319716829" watchObservedRunningTime="2026-09-17 15:45:33.850884046 +0000 UTC m=+399.322723897"
Sep 17 15:45:33.874197 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:33.874139    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8" podStartSLOduration=1.418047873 podStartE2EDuration="4.874122435s" totalImagesPullingTime="3.456074562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:33.872019887 +0000 UTC m=+399.343859734" watchObservedRunningTime="2026-09-17 15:45:33.874122435 +0000 UTC m=+399.345962277"
Sep 17 15:45:44.839973 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:44.839945    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-ksxg8"
Sep 17 15:45:49.059494 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.059460    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bc88b544-h4t9m"]
Sep 17 15:45:49.062918 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.062894    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.066422 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.066381    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 15:45:49.066639 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.066610    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 15:45:49.066744 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.066692    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 15:45:49.066807 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.066779    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 15:45:49.067106 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.067082    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 15:45:49.067499 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.067472    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 15:45:49.067666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.067584    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 15:45:49.067666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.067599    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t7h6z\""
Sep 17 15:45:49.071725 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.071690    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 15:45:49.075112 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.075073    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bc88b544-h4t9m"]
Sep 17 15:45:49.115746 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.115691    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:49.217413 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.217367    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:49.253681 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.253649    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-console-oauth-config\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.253834 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.253701    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw4mb\" (UniqueName: \"kubernetes.io/projected/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-kube-api-access-sw4mb\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.253834 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.253796    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-console-config\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.253936 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.253840    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-service-ca\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.253936 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.253913    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-oauth-serving-cert\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.254016 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.253960    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-trusted-ca-bundle\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.254016 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.253995    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-console-serving-cert\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.354999 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.354967    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw4mb\" (UniqueName: \"kubernetes.io/projected/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-kube-api-access-sw4mb\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.355191 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.355012    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-console-config\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.355191 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.355031    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-service-ca\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.355191 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.355061    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-oauth-serving-cert\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.355191 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.355089    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-trusted-ca-bundle\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.355191 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.355116    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-console-serving-cert\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.355191 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.355159    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-console-oauth-config\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.355978 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.355949    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-service-ca\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.355978 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.355970    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-console-config\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.356183 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.356039    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-oauth-serving-cert\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.356183 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.356152    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-trusted-ca-bundle\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.357850 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.357830    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-console-oauth-config\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.358202 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.358182    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-console-serving-cert\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.363007 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.362985    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw4mb\" (UniqueName: \"kubernetes.io/projected/735788a4-f575-4ddb-b1a6-c8c0cd80d7f6-kube-api-access-sw4mb\") pod \"console-6bc88b544-h4t9m\" (UID: \"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6\") " pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.373942 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.373915    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:49.651894 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.651802    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bc88b544-h4t9m"]
Sep 17 15:45:49.655107 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:45:49.655076    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735788a4_f575_4ddb_b1a6_c8c0cd80d7f6.slice/crio-64b26e620c397151493c91412c3baebdd0c36eebd10976de17c9b1d9b9379b0b WatchSource:0}: Error finding container 64b26e620c397151493c91412c3baebdd0c36eebd10976de17c9b1d9b9379b0b: Status 404 returned error can't find the container with id 64b26e620c397151493c91412c3baebdd0c36eebd10976de17c9b1d9b9379b0b
Sep 17 15:45:49.902336 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.902247    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc88b544-h4t9m" event={"ID":"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6","Type":"ContainerStarted","Data":"65c63f33665e17ee7de5d275981ed4161ff164288e63b94bb5c456d08ef05527"}
Sep 17 15:45:49.902336 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.902312    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc88b544-h4t9m" event={"ID":"735788a4-f575-4ddb-b1a6-c8c0cd80d7f6","Type":"ContainerStarted","Data":"64b26e620c397151493c91412c3baebdd0c36eebd10976de17c9b1d9b9379b0b"}
Sep 17 15:45:49.930969 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:49.930909    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bc88b544-h4t9m" podStartSLOduration=0.930890349 podStartE2EDuration="930.890349ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:49.930564178 +0000 UTC m=+415.402404041" watchObservedRunningTime="2026-09-17 15:45:49.930890349 +0000 UTC m=+415.402730194"
Sep 17 15:45:54.838934 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:54.838904    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-j5shx"
Sep 17 15:45:59.374586 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:59.374543    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:59.374942 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:59.374608    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:59.379311 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:59.379286    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:45:59.948197 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:45:59.948171    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bc88b544-h4t9m"
Sep 17 15:46:00.091269 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:00.091232    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:00.149263 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:00.149225    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:00.211134 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:00.211052    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:21.711935 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:21.711853    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:21.809393 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:21.809355    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:32.857036 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:32.856998    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:33.695173 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:33.695133    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:36.425002 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:36.424967    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:36.479243 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:36.479214    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:38.813307 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:38.813254    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:38.919028 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:38.918989    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:41.097602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:41.097565    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:41.321250 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:41.321219    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:41.450101 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:41.450060    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:41.546673 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:41.546634    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:43.175926 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:43.175889    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:43.281170 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:43.281130    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:43.375275 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:43.375243    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:46.925099 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:46.925057    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:46.987550 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:46.987516    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:48.303431 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:48.303398    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:55.057650 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:55.057607    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:55.159128 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:46:55.159094    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:06.721379 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:06.721342    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:11.513114 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.513070    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:11.534140 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.534106    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"]
Sep 17 15:47:11.537331 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.537316    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.539547 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.539521    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 15:47:11.539547 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.539541    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 15:47:11.539741 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.539523    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 15:47:11.539930 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.539918    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lsm7w\""
Sep 17 15:47:11.542494 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.542475    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 15:47:11.552079 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.552058    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"]
Sep 17 15:47:11.577198 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.577168    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:11.623342 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.623314    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.623488 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.623356    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.623488 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.623388    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.623488 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.623421    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.623488 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.623474    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp4lz\" (UniqueName: \"kubernetes.io/projected/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-kube-api-access-wp4lz\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.623640 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.623502    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.623640 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.623531    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.724350 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.724305    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp4lz\" (UniqueName: \"kubernetes.io/projected/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-kube-api-access-wp4lz\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.724552 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.724368    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.724552 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.724406    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.724552 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.724478    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.724552 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.724512    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.724552 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.724540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.724800 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.724579    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.725257 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.725228    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.726876 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.726852    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.726965 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.726882    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.727006 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.726964    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.727006 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.726990    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.732660 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.732635    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp4lz\" (UniqueName: \"kubernetes.io/projected/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-kube-api-access-wp4lz\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.733168 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.733147    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-h49fq\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:11.846410 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:11.846313    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:12.126683 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:12.126651    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"]
Sep 17 15:47:12.130158 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:47:12.130130    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7de9ab_ab16_4c85_bcd6_1adee9fb9962.slice/crio-b5b09fce8c7bef634c4da41f6510e071304b37d32275d1832edda919a5f7d82f WatchSource:0}: Error finding container b5b09fce8c7bef634c4da41f6510e071304b37d32275d1832edda919a5f7d82f: Status 404 returned error can't find the container with id b5b09fce8c7bef634c4da41f6510e071304b37d32275d1832edda919a5f7d82f
Sep 17 15:47:12.225240 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:12.225208    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq" event={"ID":"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962","Type":"ContainerStarted","Data":"b5b09fce8c7bef634c4da41f6510e071304b37d32275d1832edda919a5f7d82f"}
Sep 17 15:47:12.720320 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:12.720287    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:14.925588 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:14.925549    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 15:47:14.925947 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:14.925631    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 15:47:15.245432 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:15.245305    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq" event={"ID":"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962","Type":"ContainerStarted","Data":"43e28e1d9dbe11918ae782dfdf53f0b07572b26152ed2d73774ee620c9c4a3db"}
Sep 17 15:47:15.245658 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:15.245626    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:15.247809 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:15.247782    2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-h49fq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 15:47:15.247919 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:15.247836    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq" podUID="cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:47:15.268119 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:15.268067    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq" podStartSLOduration=1.475433442 podStartE2EDuration="4.268051561s" totalImagesPullingTime="2.792618119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:15.266338098 +0000 UTC m=+500.738177943" watchObservedRunningTime="2026-09-17 15:47:15.268051561 +0000 UTC m=+500.739891406"
Sep 17 15:47:15.862496 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:15.862465    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:15.947460 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:15.947391    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:16.250254 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:16.250178    2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-h49fq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 15:47:16.250254 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:16.250233    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq" podUID="cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:47:19.250519 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:19.250490    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:47:20.163156 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:20.163124    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:20.261461 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:20.261404    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:22.869407 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:22.869369    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:24.606282 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:24.606250    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:24.700219 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:24.700185    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:25.260643 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:25.260611    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:25.362803 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:25.362770    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:32.400287 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:32.400254    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:37.616543 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:37.616502    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:40.416862 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:40.416826    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:41.224406 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:41.224370    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:42.005642 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:42.005603    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:44.807708 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:44.807670    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq"]
Sep 17 15:47:44.812836 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:44.812815    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq"
Sep 17 15:47:44.815114 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:44.815080    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 15:47:44.815254 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:44.815080    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 15:47:44.815254 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:44.815125    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-86pfh\""
Sep 17 15:47:44.826327 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:44.826282    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq"]
Sep 17 15:47:44.906186 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:44.906146    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbgn4\" (UniqueName: \"kubernetes.io/projected/ca00977e-3c20-4628-ae3d-125aa5d1756e-kube-api-access-kbgn4\") pod \"kuadrant-operator-controller-manager-6d4cb79c-szvjq\" (UID: \"ca00977e-3c20-4628-ae3d-125aa5d1756e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq"
Sep 17 15:47:44.906353 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:44.906220    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca00977e-3c20-4628-ae3d-125aa5d1756e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-szvjq\" (UID: \"ca00977e-3c20-4628-ae3d-125aa5d1756e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq"
Sep 17 15:47:45.007518 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:45.007488    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca00977e-3c20-4628-ae3d-125aa5d1756e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-szvjq\" (UID: \"ca00977e-3c20-4628-ae3d-125aa5d1756e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq"
Sep 17 15:47:45.007693 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:45.007584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbgn4\" (UniqueName: \"kubernetes.io/projected/ca00977e-3c20-4628-ae3d-125aa5d1756e-kube-api-access-kbgn4\") pod \"kuadrant-operator-controller-manager-6d4cb79c-szvjq\" (UID: \"ca00977e-3c20-4628-ae3d-125aa5d1756e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq"
Sep 17 15:47:45.007951 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:45.007926    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca00977e-3c20-4628-ae3d-125aa5d1756e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-szvjq\" (UID: \"ca00977e-3c20-4628-ae3d-125aa5d1756e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq"
Sep 17 15:47:45.022243 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:45.022211    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbgn4\" (UniqueName: \"kubernetes.io/projected/ca00977e-3c20-4628-ae3d-125aa5d1756e-kube-api-access-kbgn4\") pod \"kuadrant-operator-controller-manager-6d4cb79c-szvjq\" (UID: \"ca00977e-3c20-4628-ae3d-125aa5d1756e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq"
Sep 17 15:47:45.048456 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:45.048401    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:45.122389 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:45.122356    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq"
Sep 17 15:47:45.409124 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:45.409097    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq"]
Sep 17 15:47:45.411971 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:47:45.411940    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca00977e_3c20_4628_ae3d_125aa5d1756e.slice/crio-a282275336e43b2f8862dcd481f6f84b177f34bec62c603b93328a40f0153148 WatchSource:0}: Error finding container a282275336e43b2f8862dcd481f6f84b177f34bec62c603b93328a40f0153148: Status 404 returned error can't find the container with id a282275336e43b2f8862dcd481f6f84b177f34bec62c603b93328a40f0153148
Sep 17 15:47:45.426718 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:45.426678    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:46.148859 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:46.148818    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:46.249340 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:46.249307    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:46.365891 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:46.365843    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq" event={"ID":"ca00977e-3c20-4628-ae3d-125aa5d1756e","Type":"ContainerStarted","Data":"a282275336e43b2f8862dcd481f6f84b177f34bec62c603b93328a40f0153148"}
Sep 17 15:47:47.280232 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:47.280071    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-26g8k"]
Sep 17 15:47:47.282705 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:47.282682    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-26g8k"
Sep 17 15:47:47.285058 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:47.285011    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ngc52\""
Sep 17 15:47:47.296186 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:47.296156    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-26g8k"]
Sep 17 15:47:47.428183 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:47.428140    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt27d\" (UniqueName: \"kubernetes.io/projected/d06c1525-0f0c-4b12-b6ef-9b2a297400af-kube-api-access-wt27d\") pod \"authorino-operator-6fc8f864f4-26g8k\" (UID: \"d06c1525-0f0c-4b12-b6ef-9b2a297400af\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-26g8k"
Sep 17 15:47:47.529193 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:47.529154    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt27d\" (UniqueName: \"kubernetes.io/projected/d06c1525-0f0c-4b12-b6ef-9b2a297400af-kube-api-access-wt27d\") pod \"authorino-operator-6fc8f864f4-26g8k\" (UID: \"d06c1525-0f0c-4b12-b6ef-9b2a297400af\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-26g8k"
Sep 17 15:47:47.538863 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:47.538797    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt27d\" (UniqueName: \"kubernetes.io/projected/d06c1525-0f0c-4b12-b6ef-9b2a297400af-kube-api-access-wt27d\") pod \"authorino-operator-6fc8f864f4-26g8k\" (UID: \"d06c1525-0f0c-4b12-b6ef-9b2a297400af\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-26g8k"
Sep 17 15:47:47.595634 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:47.595595    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-26g8k"
Sep 17 15:47:47.667198 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:47.667141    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:47.770039 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:47.769985    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:47.861026 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:47.860991    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:47.878039 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:47.878005    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-26g8k"]
Sep 17 15:47:47.883655 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:47:47.883618    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06c1525_0f0c_4b12_b6ef_9b2a297400af.slice/crio-3abef6951949677086e2d747d5db8b48247afe967fc5f084d0dd8af2a961f905 WatchSource:0}: Error finding container 3abef6951949677086e2d747d5db8b48247afe967fc5f084d0dd8af2a961f905: Status 404 returned error can't find the container with id 3abef6951949677086e2d747d5db8b48247afe967fc5f084d0dd8af2a961f905
Sep 17 15:47:48.386635 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:48.386595    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-26g8k" event={"ID":"d06c1525-0f0c-4b12-b6ef-9b2a297400af","Type":"ContainerStarted","Data":"3abef6951949677086e2d747d5db8b48247afe967fc5f084d0dd8af2a961f905"}
Sep 17 15:47:49.394802 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:49.394734    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq" event={"ID":"ca00977e-3c20-4628-ae3d-125aa5d1756e","Type":"ContainerStarted","Data":"2445ab301af6223ab1c049d334428cf4535162d607eac5d5435ed7a8c0a968ff"}
Sep 17 15:47:49.395219 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:49.394907    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq"
Sep 17 15:47:49.403808 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:49.403782    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qxkqk"]
Sep 17 15:47:49.406197 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:49.406177    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qxkqk"
Sep 17 15:47:49.408206 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:49.408184    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7bzk6\""
Sep 17 15:47:49.421653 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:49.420406    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq" podStartSLOduration=1.585761384 podStartE2EDuration="5.420384675s" totalImagesPullingTime="3.834623291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:49.418544866 +0000 UTC m=+534.890384724" watchObservedRunningTime="2026-09-17 15:47:49.420384675 +0000 UTC m=+534.892224524"
Sep 17 15:47:49.423388 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:49.423362    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qxkqk"]
Sep 17 15:47:49.550119 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:49.550014    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds7g2\" (UniqueName: \"kubernetes.io/projected/e5f51b66-517d-4073-82ad-95511a0cca5d-kube-api-access-ds7g2\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-qxkqk\" (UID: \"e5f51b66-517d-4073-82ad-95511a0cca5d\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qxkqk"
Sep 17 15:47:49.651624 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:49.651590    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds7g2\" (UniqueName: \"kubernetes.io/projected/e5f51b66-517d-4073-82ad-95511a0cca5d-kube-api-access-ds7g2\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-qxkqk\" (UID: \"e5f51b66-517d-4073-82ad-95511a0cca5d\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qxkqk"
Sep 17 15:47:49.665527 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:49.665491    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds7g2\" (UniqueName: \"kubernetes.io/projected/e5f51b66-517d-4073-82ad-95511a0cca5d-kube-api-access-ds7g2\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-qxkqk\" (UID: \"e5f51b66-517d-4073-82ad-95511a0cca5d\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qxkqk"
Sep 17 15:47:49.720339 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:49.720306    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qxkqk"
Sep 17 15:47:49.883572 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:49.883529    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:50.008062 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:50.008007    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:50.201142 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:50.201117    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qxkqk"]
Sep 17 15:47:50.204160 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:47:50.204131    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f51b66_517d_4073_82ad_95511a0cca5d.slice/crio-f7154bdc01a45dcd56bd92aab30047eec4230d88947e346777dfd6b62088fbfc WatchSource:0}: Error finding container f7154bdc01a45dcd56bd92aab30047eec4230d88947e346777dfd6b62088fbfc: Status 404 returned error can't find the container with id f7154bdc01a45dcd56bd92aab30047eec4230d88947e346777dfd6b62088fbfc
Sep 17 15:47:50.400522 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:50.400485    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qxkqk" event={"ID":"e5f51b66-517d-4073-82ad-95511a0cca5d","Type":"ContainerStarted","Data":"f7154bdc01a45dcd56bd92aab30047eec4230d88947e346777dfd6b62088fbfc"}
Sep 17 15:47:50.402251 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:50.402223    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-26g8k" event={"ID":"d06c1525-0f0c-4b12-b6ef-9b2a297400af","Type":"ContainerStarted","Data":"3fabd9b971f20ad1785781bfd57d996c6e955a87fc416aceccd9dab0e9b64640"}
Sep 17 15:47:50.420335 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:50.420286    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-26g8k" podStartSLOduration=1.340827523 podStartE2EDuration="3.420271549s" totalImagesPullingTime="2.079444026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:50.419751762 +0000 UTC m=+535.891591609" watchObservedRunningTime="2026-09-17 15:47:50.420271549 +0000 UTC m=+535.892111394"
Sep 17 15:47:51.406993 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:51.406963    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-26g8k"
Sep 17 15:47:51.953135 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:51.953092    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:52.074138 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:52.074037    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:52.412746 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:52.412694    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qxkqk" event={"ID":"e5f51b66-517d-4073-82ad-95511a0cca5d","Type":"ContainerStarted","Data":"e557803eb107989f7bc239197c3ff6adb5ff6ef69fa153ab10d9eeacffe81472"}
Sep 17 15:47:52.433718 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:52.433659    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qxkqk" podStartSLOduration=1.3391082939999999 podStartE2EDuration="3.433641785s" totalImagesPullingTime="2.094533491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:52.431657194 +0000 UTC m=+537.903497078" watchObservedRunningTime="2026-09-17 15:47:52.433641785 +0000 UTC m=+537.905481631"
Sep 17 15:47:52.471729 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:52.471697    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:53.417681 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:53.417652    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qxkqk"
Sep 17 15:47:55.402390 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:55.402356    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:55.503727 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:55.503692    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:56.212113 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:56.212086    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:56.312345 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:56.312308    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:56.610025 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:56.609993    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:56.709928 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:47:56.709888    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:00.405206 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:00.405171    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-szvjq"
Sep 17 15:48:02.415182 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:02.415150    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-26g8k"
Sep 17 15:48:04.425320 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:04.425281    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qxkqk"
Sep 17 15:48:04.902310 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:04.902270    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:04.987540 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:04.987502    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:07.824638 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:07.824604    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:07.926501 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:07.926468    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:10.273053 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:10.273015    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:10.374562 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:10.374528    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:17.234230 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:17.234193    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:17.333897 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:17.333862    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:18.689132 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:18.689096    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:18.794694 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:18.794650    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:27.238266 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:27.238230    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:27.339318 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:27.339280    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:27.823501 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:27.823465    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:27.875977 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:27.875935    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:31.637184 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:31.637146    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:31.738498 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:31.738464    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:32.857423 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:32.857371    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:32.955244 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:32.955206    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:33.278518 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.278475    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-892hx"]
Sep 17 15:48:33.280981 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.280956    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-892hx"
Sep 17 15:48:33.284626 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.284600    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w6hk6\""
Sep 17 15:48:33.284626 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.284618    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 15:48:33.297845 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.297814    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-892hx"]
Sep 17 15:48:33.380955 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.380915    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-892hx"]
Sep 17 15:48:33.424750 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.424641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1facedd8-3eae-45ce-9516-21618b921ea5-config-file\") pod \"limitador-limitador-84994f7ffc-892hx\" (UID: \"1facedd8-3eae-45ce-9516-21618b921ea5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-892hx"
Sep 17 15:48:33.424750 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.424713    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r49ng\" (UniqueName: \"kubernetes.io/projected/1facedd8-3eae-45ce-9516-21618b921ea5-kube-api-access-r49ng\") pod \"limitador-limitador-84994f7ffc-892hx\" (UID: \"1facedd8-3eae-45ce-9516-21618b921ea5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-892hx"
Sep 17 15:48:33.526113 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.526073    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1facedd8-3eae-45ce-9516-21618b921ea5-config-file\") pod \"limitador-limitador-84994f7ffc-892hx\" (UID: \"1facedd8-3eae-45ce-9516-21618b921ea5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-892hx"
Sep 17 15:48:33.526294 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.526126    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49ng\" (UniqueName: \"kubernetes.io/projected/1facedd8-3eae-45ce-9516-21618b921ea5-kube-api-access-r49ng\") pod \"limitador-limitador-84994f7ffc-892hx\" (UID: \"1facedd8-3eae-45ce-9516-21618b921ea5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-892hx"
Sep 17 15:48:33.526812 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.526790    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1facedd8-3eae-45ce-9516-21618b921ea5-config-file\") pod \"limitador-limitador-84994f7ffc-892hx\" (UID: \"1facedd8-3eae-45ce-9516-21618b921ea5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-892hx"
Sep 17 15:48:33.564548 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.564525    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r49ng\" (UniqueName: \"kubernetes.io/projected/1facedd8-3eae-45ce-9516-21618b921ea5-kube-api-access-r49ng\") pod \"limitador-limitador-84994f7ffc-892hx\" (UID: \"1facedd8-3eae-45ce-9516-21618b921ea5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-892hx"
Sep 17 15:48:33.591144 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.591111    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-892hx"
Sep 17 15:48:33.868640 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.868606    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-892hx"]
Sep 17 15:48:33.873363 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:48:33.873336    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1facedd8_3eae_45ce_9516_21618b921ea5.slice/crio-5c367e260e1283d247efef07eeaf90abc9cee1249f8a1826bf71ee28e4773792 WatchSource:0}: Error finding container 5c367e260e1283d247efef07eeaf90abc9cee1249f8a1826bf71ee28e4773792: Status 404 returned error can't find the container with id 5c367e260e1283d247efef07eeaf90abc9cee1249f8a1826bf71ee28e4773792
Sep 17 15:48:33.997287 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:33.997244    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:34.072706 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:34.072672    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:34.437010 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:34.436975    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:34.533970 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:34.533933    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:34.604353 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:34.604317    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-892hx" event={"ID":"1facedd8-3eae-45ce-9516-21618b921ea5","Type":"ContainerStarted","Data":"5c367e260e1283d247efef07eeaf90abc9cee1249f8a1826bf71ee28e4773792"}
Sep 17 15:48:35.610708 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:35.610669    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-892hx" event={"ID":"1facedd8-3eae-45ce-9516-21618b921ea5","Type":"ContainerStarted","Data":"55c7c2a97dcda16106a620e581f84f0e3459820b2056ef9d83f9e4e3bbf4dbe3"}
Sep 17 15:48:35.611181 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:35.610902    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-892hx"
Sep 17 15:48:35.628018 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:35.627961    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-892hx" podStartSLOduration=1.41996588 podStartE2EDuration="2.627940963s" totalImagesPullingTime="1.207975083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:48:35.626771052 +0000 UTC m=+581.098610898" watchObservedRunningTime="2026-09-17 15:48:35.627940963 +0000 UTC m=+581.099780810"
Sep 17 15:48:37.486885 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:37.486846    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:37.589513 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:37.589475    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:41.227415 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:41.227380    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:41.286118 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:41.286082    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:41.434471 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:41.434396    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:41.513901 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:41.513810    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:42.429181 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:42.429118    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:42.531465 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:42.531413    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:46.616784 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:46.616746    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-892hx"
Sep 17 15:48:49.841467 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:49.841402    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:49.942331 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:49.942276    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:52.803169 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:52.803130    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:52.913814 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:52.913778    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:53.038984 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:53.038946    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:53.451040 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:53.450999    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:55.346969 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:55.346927    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:55.446043 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:55.446002    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:55.510433 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:48:55.510386    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:01.831129 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:01.831095    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:01.935755 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:01.935727    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:06.060311 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:06.060244    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:10.611105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:10.611064    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:14.735326 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:14.735289    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:20.408353 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.408277    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"]
Sep 17 15:49:20.408767 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.408480    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq" podUID="cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" containerName="discovery" containerID="cri-o://43e28e1d9dbe11918ae782dfdf53f0b07572b26152ed2d73774ee620c9c4a3db" gracePeriod=30
Sep 17 15:49:20.652035 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.652010    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:49:20.740959 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.740879    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-cacerts\") pod \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") "
Sep 17 15:49:20.740959 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.740916    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp4lz\" (UniqueName: \"kubernetes.io/projected/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-kube-api-access-wp4lz\") pod \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") "
Sep 17 15:49:20.740959 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.740961    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-csr-dns-cert\") pod \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") "
Sep 17 15:49:20.741227 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.740979    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-token\") pod \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") "
Sep 17 15:49:20.741227 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.741021    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-csr-ca-configmap\") pod \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") "
Sep 17 15:49:20.741227 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.741052    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-kubeconfig\") pod \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") "
Sep 17 15:49:20.741227 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.741102    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-local-certs\") pod \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\" (UID: \"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962\") "
Sep 17 15:49:20.741528 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.741497    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" (UID: "cd7de9ab-ab16-4c85-bcd6-1adee9fb9962"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:49:20.743553 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.743521    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" (UID: "cd7de9ab-ab16-4c85-bcd6-1adee9fb9962"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:49:20.743648 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.743552    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-cacerts" (OuterVolumeSpecName: "cacerts") pod "cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" (UID: "cd7de9ab-ab16-4c85-bcd6-1adee9fb9962"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:49:20.743648 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.743569    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-local-certs" (OuterVolumeSpecName: "local-certs") pod "cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" (UID: "cd7de9ab-ab16-4c85-bcd6-1adee9fb9962"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:20.743648 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.743596    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" (UID: "cd7de9ab-ab16-4c85-bcd6-1adee9fb9962"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:49:20.743762 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.743648    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" (UID: "cd7de9ab-ab16-4c85-bcd6-1adee9fb9962"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:49:20.743762 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.743653    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-kube-api-access-wp4lz" (OuterVolumeSpecName: "kube-api-access-wp4lz") pod "cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" (UID: "cd7de9ab-ab16-4c85-bcd6-1adee9fb9962"). InnerVolumeSpecName "kube-api-access-wp4lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:49:20.813360 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.813322    2574 generic.go:356] "Generic (PLEG): container finished" podID="cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" containerID="43e28e1d9dbe11918ae782dfdf53f0b07572b26152ed2d73774ee620c9c4a3db" exitCode=0
Sep 17 15:49:20.813562 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.813371    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"
Sep 17 15:49:20.813562 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.813410    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq" event={"ID":"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962","Type":"ContainerDied","Data":"43e28e1d9dbe11918ae782dfdf53f0b07572b26152ed2d73774ee620c9c4a3db"}
Sep 17 15:49:20.813562 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.813491    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq" event={"ID":"cd7de9ab-ab16-4c85-bcd6-1adee9fb9962","Type":"ContainerDied","Data":"b5b09fce8c7bef634c4da41f6510e071304b37d32275d1832edda919a5f7d82f"}
Sep 17 15:49:20.813562 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.813513    2574 scope.go:117] "RemoveContainer" containerID="43e28e1d9dbe11918ae782dfdf53f0b07572b26152ed2d73774ee620c9c4a3db"
Sep 17 15:49:20.824338 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.824318    2574 scope.go:117] "RemoveContainer" containerID="43e28e1d9dbe11918ae782dfdf53f0b07572b26152ed2d73774ee620c9c4a3db"
Sep 17 15:49:20.824699 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:49:20.824667    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e28e1d9dbe11918ae782dfdf53f0b07572b26152ed2d73774ee620c9c4a3db\": container with ID starting with 43e28e1d9dbe11918ae782dfdf53f0b07572b26152ed2d73774ee620c9c4a3db not found: ID does not exist" containerID="43e28e1d9dbe11918ae782dfdf53f0b07572b26152ed2d73774ee620c9c4a3db"
Sep 17 15:49:20.824800 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.824698    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e28e1d9dbe11918ae782dfdf53f0b07572b26152ed2d73774ee620c9c4a3db"} err="failed to get container status \"43e28e1d9dbe11918ae782dfdf53f0b07572b26152ed2d73774ee620c9c4a3db\": rpc error: code = NotFound desc = could not find container \"43e28e1d9dbe11918ae782dfdf53f0b07572b26152ed2d73774ee620c9c4a3db\": container with ID starting with 43e28e1d9dbe11918ae782dfdf53f0b07572b26152ed2d73774ee620c9c4a3db not found: ID does not exist"
Sep 17 15:49:20.838103 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.838072    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"]
Sep 17 15:49:20.841165 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.841143    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h49fq"]
Sep 17 15:49:20.841881 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.841864    2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-csr-ca-configmap\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:49:20.841940 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.841884    2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-kubeconfig\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:49:20.841940 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.841895    2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-local-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:49:20.841940 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.841903    2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-cacerts\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:49:20.841940 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.841911    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp4lz\" (UniqueName: \"kubernetes.io/projected/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-kube-api-access-wp4lz\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:49:20.841940 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.841921    2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-csr-dns-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:49:20.841940 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:20.841930    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962-istio-token\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:49:21.048744 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:21.048650    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" path="/var/lib/kubelet/pods/cd7de9ab-ab16-4c85-bcd6-1adee9fb9962/volumes"
Sep 17 15:49:30.334963 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.334922    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:30.410699 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.410632    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:30.414776 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.414748    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-74b7f9c78b-6hwz6"]
Sep 17 15:49:30.415160 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.415147    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" containerName="discovery"
Sep 17 15:49:30.415206 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.415161    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" containerName="discovery"
Sep 17 15:49:30.415247 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.415216    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd7de9ab-ab16-4c85-bcd6-1adee9fb9962" containerName="discovery"
Sep 17 15:49:30.419430 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.419414    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6"
Sep 17 15:49:30.421674 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.421646    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-tvpn6\""
Sep 17 15:49:30.421780 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.421685    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 15:49:30.421780 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.421682    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 15:49:30.422121 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.422108    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 15:49:30.425711 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.425689    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-74b7f9c78b-6hwz6"]
Sep 17 15:49:30.449037 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.449011    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-745575db98-qjzqx"]
Sep 17 15:49:30.452238 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.452219    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-745575db98-qjzqx"
Sep 17 15:49:30.454240 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.454217    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 15:49:30.454360 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.454341    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-zmx62\""
Sep 17 15:49:30.461179 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.461156    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-745575db98-qjzqx"]
Sep 17 15:49:30.467596 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.467577    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-2dgms"]
Sep 17 15:49:30.470978 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.470957    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2dgms"
Sep 17 15:49:30.473040 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.473022    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 15:49:30.473139 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.473118    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mg8nn\""
Sep 17 15:49:30.481650 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.481629    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2dgms"]
Sep 17 15:49:30.519378 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.519353    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33f0b0fb-64fa-4e36-943a-07efcc62da86-cert\") pod \"kserve-controller-manager-74b7f9c78b-6hwz6\" (UID: \"33f0b0fb-64fa-4e36-943a-07efcc62da86\") " pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6"
Sep 17 15:49:30.519533 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.519382    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp5pt\" (UniqueName: \"kubernetes.io/projected/33f0b0fb-64fa-4e36-943a-07efcc62da86-kube-api-access-rp5pt\") pod \"kserve-controller-manager-74b7f9c78b-6hwz6\" (UID: \"33f0b0fb-64fa-4e36-943a-07efcc62da86\") " pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6"
Sep 17 15:49:30.561217 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.561177    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:30.620868 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.620839    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d272397f-264e-443b-9298-ee3a3ecbd9ef-data\") pod \"seaweedfs-64568bcd49-2dgms\" (UID: \"d272397f-264e-443b-9298-ee3a3ecbd9ef\") " pod="kserve/seaweedfs-64568bcd49-2dgms"
Sep 17 15:49:30.621041 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.620922    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33f0b0fb-64fa-4e36-943a-07efcc62da86-cert\") pod \"kserve-controller-manager-74b7f9c78b-6hwz6\" (UID: \"33f0b0fb-64fa-4e36-943a-07efcc62da86\") " pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6"
Sep 17 15:49:30.621041 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.620945    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp5pt\" (UniqueName: \"kubernetes.io/projected/33f0b0fb-64fa-4e36-943a-07efcc62da86-kube-api-access-rp5pt\") pod \"kserve-controller-manager-74b7f9c78b-6hwz6\" (UID: \"33f0b0fb-64fa-4e36-943a-07efcc62da86\") " pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6"
Sep 17 15:49:30.621041 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.620962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tphw\" (UniqueName: \"kubernetes.io/projected/d272397f-264e-443b-9298-ee3a3ecbd9ef-kube-api-access-6tphw\") pod \"seaweedfs-64568bcd49-2dgms\" (UID: \"d272397f-264e-443b-9298-ee3a3ecbd9ef\") " pod="kserve/seaweedfs-64568bcd49-2dgms"
Sep 17 15:49:30.621041 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.621001    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csxmz\" (UniqueName: \"kubernetes.io/projected/b708ab85-a7dd-478d-af39-33b54278e0f7-kube-api-access-csxmz\") pod \"llmisvc-controller-manager-745575db98-qjzqx\" (UID: \"b708ab85-a7dd-478d-af39-33b54278e0f7\") " pod="kserve/llmisvc-controller-manager-745575db98-qjzqx"
Sep 17 15:49:30.621222 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.621085    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b708ab85-a7dd-478d-af39-33b54278e0f7-cert\") pod \"llmisvc-controller-manager-745575db98-qjzqx\" (UID: \"b708ab85-a7dd-478d-af39-33b54278e0f7\") " pod="kserve/llmisvc-controller-manager-745575db98-qjzqx"
Sep 17 15:49:30.623454 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.623419    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33f0b0fb-64fa-4e36-943a-07efcc62da86-cert\") pod \"kserve-controller-manager-74b7f9c78b-6hwz6\" (UID: \"33f0b0fb-64fa-4e36-943a-07efcc62da86\") " pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6"
Sep 17 15:49:30.631369 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.631347    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp5pt\" (UniqueName: \"kubernetes.io/projected/33f0b0fb-64fa-4e36-943a-07efcc62da86-kube-api-access-rp5pt\") pod \"kserve-controller-manager-74b7f9c78b-6hwz6\" (UID: \"33f0b0fb-64fa-4e36-943a-07efcc62da86\") " pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6"
Sep 17 15:49:30.662730 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.662699    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:30.721836 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.721794    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tphw\" (UniqueName: \"kubernetes.io/projected/d272397f-264e-443b-9298-ee3a3ecbd9ef-kube-api-access-6tphw\") pod \"seaweedfs-64568bcd49-2dgms\" (UID: \"d272397f-264e-443b-9298-ee3a3ecbd9ef\") " pod="kserve/seaweedfs-64568bcd49-2dgms"
Sep 17 15:49:30.722000 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.721845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csxmz\" (UniqueName: \"kubernetes.io/projected/b708ab85-a7dd-478d-af39-33b54278e0f7-kube-api-access-csxmz\") pod \"llmisvc-controller-manager-745575db98-qjzqx\" (UID: \"b708ab85-a7dd-478d-af39-33b54278e0f7\") " pod="kserve/llmisvc-controller-manager-745575db98-qjzqx"
Sep 17 15:49:30.722000 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.721878    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b708ab85-a7dd-478d-af39-33b54278e0f7-cert\") pod \"llmisvc-controller-manager-745575db98-qjzqx\" (UID: \"b708ab85-a7dd-478d-af39-33b54278e0f7\") " pod="kserve/llmisvc-controller-manager-745575db98-qjzqx"
Sep 17 15:49:30.722000 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.721898    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d272397f-264e-443b-9298-ee3a3ecbd9ef-data\") pod \"seaweedfs-64568bcd49-2dgms\" (UID: \"d272397f-264e-443b-9298-ee3a3ecbd9ef\") " pod="kserve/seaweedfs-64568bcd49-2dgms"
Sep 17 15:49:30.722279 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.722262    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d272397f-264e-443b-9298-ee3a3ecbd9ef-data\") pod \"seaweedfs-64568bcd49-2dgms\" (UID: \"d272397f-264e-443b-9298-ee3a3ecbd9ef\") " pod="kserve/seaweedfs-64568bcd49-2dgms"
Sep 17 15:49:30.724271 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.724247    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b708ab85-a7dd-478d-af39-33b54278e0f7-cert\") pod \"llmisvc-controller-manager-745575db98-qjzqx\" (UID: \"b708ab85-a7dd-478d-af39-33b54278e0f7\") " pod="kserve/llmisvc-controller-manager-745575db98-qjzqx"
Sep 17 15:49:30.730300 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.730275    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6"
Sep 17 15:49:30.730526 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.730506    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csxmz\" (UniqueName: \"kubernetes.io/projected/b708ab85-a7dd-478d-af39-33b54278e0f7-kube-api-access-csxmz\") pod \"llmisvc-controller-manager-745575db98-qjzqx\" (UID: \"b708ab85-a7dd-478d-af39-33b54278e0f7\") " pod="kserve/llmisvc-controller-manager-745575db98-qjzqx"
Sep 17 15:49:30.730714 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.730697    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tphw\" (UniqueName: \"kubernetes.io/projected/d272397f-264e-443b-9298-ee3a3ecbd9ef-kube-api-access-6tphw\") pod \"seaweedfs-64568bcd49-2dgms\" (UID: \"d272397f-264e-443b-9298-ee3a3ecbd9ef\") " pod="kserve/seaweedfs-64568bcd49-2dgms"
Sep 17 15:49:30.762560 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.762527    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-745575db98-qjzqx"
Sep 17 15:49:30.782654 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.782300    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2dgms"
Sep 17 15:49:30.898554 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:30.898465    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:31.015738 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:31.015709    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-74b7f9c78b-6hwz6"]
Sep 17 15:49:31.018127 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:49:31.018098    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f0b0fb_64fa_4e36_943a_07efcc62da86.slice/crio-76a4cf25bca892713ca0a4ad9f48c288fe55ca2f74974ed71ac506105bb9a8f2 WatchSource:0}: Error finding container 76a4cf25bca892713ca0a4ad9f48c288fe55ca2f74974ed71ac506105bb9a8f2: Status 404 returned error can't find the container with id 76a4cf25bca892713ca0a4ad9f48c288fe55ca2f74974ed71ac506105bb9a8f2
Sep 17 15:49:31.018265 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:31.018228    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:31.064370 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:31.064347    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-745575db98-qjzqx"]
Sep 17 15:49:31.066177 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:49:31.066155    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb708ab85_a7dd_478d_af39_33b54278e0f7.slice/crio-50004e93ea20b39c5de56f757fcbc48506a4afa2bb856cacb17f24ded2906c5b WatchSource:0}: Error finding container 50004e93ea20b39c5de56f757fcbc48506a4afa2bb856cacb17f24ded2906c5b: Status 404 returned error can't find the container with id 50004e93ea20b39c5de56f757fcbc48506a4afa2bb856cacb17f24ded2906c5b
Sep 17 15:49:31.096683 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:31.096657    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2dgms"]
Sep 17 15:49:31.098422 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:49:31.098399    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd272397f_264e_443b_9298_ee3a3ecbd9ef.slice/crio-b2576ebedcf8eb8275aec80745ccc4cfb6b030dd71307b8a4840291b4c4b0eb3 WatchSource:0}: Error finding container b2576ebedcf8eb8275aec80745ccc4cfb6b030dd71307b8a4840291b4c4b0eb3: Status 404 returned error can't find the container with id b2576ebedcf8eb8275aec80745ccc4cfb6b030dd71307b8a4840291b4c4b0eb3
Sep 17 15:49:31.138010 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:31.137975    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:31.217675 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:31.217590    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:31.867166 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:31.867086    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2dgms" event={"ID":"d272397f-264e-443b-9298-ee3a3ecbd9ef","Type":"ContainerStarted","Data":"b2576ebedcf8eb8275aec80745ccc4cfb6b030dd71307b8a4840291b4c4b0eb3"}
Sep 17 15:49:31.868834 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:31.868800    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-745575db98-qjzqx" event={"ID":"b708ab85-a7dd-478d-af39-33b54278e0f7","Type":"ContainerStarted","Data":"50004e93ea20b39c5de56f757fcbc48506a4afa2bb856cacb17f24ded2906c5b"}
Sep 17 15:49:31.870417 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:31.870383    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6" event={"ID":"33f0b0fb-64fa-4e36-943a-07efcc62da86","Type":"ContainerStarted","Data":"76a4cf25bca892713ca0a4ad9f48c288fe55ca2f74974ed71ac506105bb9a8f2"}
Sep 17 15:49:35.898862 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:35.898829    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2dgms" event={"ID":"d272397f-264e-443b-9298-ee3a3ecbd9ef","Type":"ContainerStarted","Data":"50468e2cef09ad0224778cd067e92e9ff04cfdd4f6bc0b98ed8c05b5571c87bc"}
Sep 17 15:49:35.899243 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:35.898997    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-2dgms"
Sep 17 15:49:35.900679 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:35.900646    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-745575db98-qjzqx" event={"ID":"b708ab85-a7dd-478d-af39-33b54278e0f7","Type":"ContainerStarted","Data":"836b2baf23378910bff2ac94219081d25a8122ee4f446217c59b4233974a1681"}
Sep 17 15:49:35.900777 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:35.900741    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-745575db98-qjzqx"
Sep 17 15:49:35.902236 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:35.902215    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6" event={"ID":"33f0b0fb-64fa-4e36-943a-07efcc62da86","Type":"ContainerStarted","Data":"a26f9ee201f6e9060f1aee954f402528acd12c0bb5beca59bdef7a872b0f55c8"}
Sep 17 15:49:35.902343 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:35.902284    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6"
Sep 17 15:49:35.921690 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:35.921629    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-2dgms" podStartSLOduration=1.227142543 podStartE2EDuration="5.921607544s" totalImagesPullingTime="4.694465001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:35.920640539 +0000 UTC m=+641.392480387" watchObservedRunningTime="2026-09-17 15:49:35.921607544 +0000 UTC m=+641.393447391"
Sep 17 15:49:35.943836 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:35.943787    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6" podStartSLOduration=1.299461959 podStartE2EDuration="5.943771078s" totalImagesPullingTime="4.644309119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:35.942558687 +0000 UTC m=+641.414398534" watchObservedRunningTime="2026-09-17 15:49:35.943771078 +0000 UTC m=+641.415610923"
Sep 17 15:49:35.967154 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:35.967091    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-745575db98-qjzqx" podStartSLOduration=1.445615272 podStartE2EDuration="5.967076058s" totalImagesPullingTime="4.521460786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:35.96454863 +0000 UTC m=+641.436388475" watchObservedRunningTime="2026-09-17 15:49:35.967076058 +0000 UTC m=+641.438915904"
Sep 17 15:49:41.910103 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:49:41.910071    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-2dgms"
Sep 17 15:50:06.909634 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:06.909601    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-745575db98-qjzqx"
Sep 17 15:50:06.912773 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:06.912746    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6"
Sep 17 15:50:15.186414 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.186375    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-74b7f9c78b-6hwz6"]
Sep 17 15:50:15.186838 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.186597    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6" podUID="33f0b0fb-64fa-4e36-943a-07efcc62da86" containerName="manager" containerID="cri-o://a26f9ee201f6e9060f1aee954f402528acd12c0bb5beca59bdef7a872b0f55c8" gracePeriod=10
Sep 17 15:50:15.212851 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.212823    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-74b7f9c78b-78vlf"]
Sep 17 15:50:15.215131 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.215115    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-74b7f9c78b-78vlf"
Sep 17 15:50:15.226363 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.226340    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-74b7f9c78b-78vlf"]
Sep 17 15:50:15.317388 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.317356    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5fbh\" (UniqueName: \"kubernetes.io/projected/299537d9-a2d7-4dd2-ae91-1e72e66b69f7-kube-api-access-z5fbh\") pod \"kserve-controller-manager-74b7f9c78b-78vlf\" (UID: \"299537d9-a2d7-4dd2-ae91-1e72e66b69f7\") " pod="kserve/kserve-controller-manager-74b7f9c78b-78vlf"
Sep 17 15:50:15.317549 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.317399    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/299537d9-a2d7-4dd2-ae91-1e72e66b69f7-cert\") pod \"kserve-controller-manager-74b7f9c78b-78vlf\" (UID: \"299537d9-a2d7-4dd2-ae91-1e72e66b69f7\") " pod="kserve/kserve-controller-manager-74b7f9c78b-78vlf"
Sep 17 15:50:15.418785 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.418755    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5fbh\" (UniqueName: \"kubernetes.io/projected/299537d9-a2d7-4dd2-ae91-1e72e66b69f7-kube-api-access-z5fbh\") pod \"kserve-controller-manager-74b7f9c78b-78vlf\" (UID: \"299537d9-a2d7-4dd2-ae91-1e72e66b69f7\") " pod="kserve/kserve-controller-manager-74b7f9c78b-78vlf"
Sep 17 15:50:15.418894 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.418797    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/299537d9-a2d7-4dd2-ae91-1e72e66b69f7-cert\") pod \"kserve-controller-manager-74b7f9c78b-78vlf\" (UID: \"299537d9-a2d7-4dd2-ae91-1e72e66b69f7\") " pod="kserve/kserve-controller-manager-74b7f9c78b-78vlf"
Sep 17 15:50:15.421200 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.421177    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/299537d9-a2d7-4dd2-ae91-1e72e66b69f7-cert\") pod \"kserve-controller-manager-74b7f9c78b-78vlf\" (UID: \"299537d9-a2d7-4dd2-ae91-1e72e66b69f7\") " pod="kserve/kserve-controller-manager-74b7f9c78b-78vlf"
Sep 17 15:50:15.423474 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.423456    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6"
Sep 17 15:50:15.427155 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.427128    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5fbh\" (UniqueName: \"kubernetes.io/projected/299537d9-a2d7-4dd2-ae91-1e72e66b69f7-kube-api-access-z5fbh\") pod \"kserve-controller-manager-74b7f9c78b-78vlf\" (UID: \"299537d9-a2d7-4dd2-ae91-1e72e66b69f7\") " pod="kserve/kserve-controller-manager-74b7f9c78b-78vlf"
Sep 17 15:50:15.520062 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.519984    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33f0b0fb-64fa-4e36-943a-07efcc62da86-cert\") pod \"33f0b0fb-64fa-4e36-943a-07efcc62da86\" (UID: \"33f0b0fb-64fa-4e36-943a-07efcc62da86\") "
Sep 17 15:50:15.520203 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.520063    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp5pt\" (UniqueName: \"kubernetes.io/projected/33f0b0fb-64fa-4e36-943a-07efcc62da86-kube-api-access-rp5pt\") pod \"33f0b0fb-64fa-4e36-943a-07efcc62da86\" (UID: \"33f0b0fb-64fa-4e36-943a-07efcc62da86\") "
Sep 17 15:50:15.522148 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.522120    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f0b0fb-64fa-4e36-943a-07efcc62da86-kube-api-access-rp5pt" (OuterVolumeSpecName: "kube-api-access-rp5pt") pod "33f0b0fb-64fa-4e36-943a-07efcc62da86" (UID: "33f0b0fb-64fa-4e36-943a-07efcc62da86"). InnerVolumeSpecName "kube-api-access-rp5pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:50:15.522148 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.522133    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33f0b0fb-64fa-4e36-943a-07efcc62da86-cert" (OuterVolumeSpecName: "cert") pod "33f0b0fb-64fa-4e36-943a-07efcc62da86" (UID: "33f0b0fb-64fa-4e36-943a-07efcc62da86"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:50:15.569470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.569418    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-74b7f9c78b-78vlf"
Sep 17 15:50:15.621252 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.621145    2574 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33f0b0fb-64fa-4e36-943a-07efcc62da86-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:50:15.621252 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.621179    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp5pt\" (UniqueName: \"kubernetes.io/projected/33f0b0fb-64fa-4e36-943a-07efcc62da86-kube-api-access-rp5pt\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:50:15.837949 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:15.837875    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-74b7f9c78b-78vlf"]
Sep 17 15:50:15.840821 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:50:15.840792    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299537d9_a2d7_4dd2_ae91_1e72e66b69f7.slice/crio-46ba2c47bd37afc3e79c3ce7775c23577925d2ef1ea0306dd4aef713e707149b WatchSource:0}: Error finding container 46ba2c47bd37afc3e79c3ce7775c23577925d2ef1ea0306dd4aef713e707149b: Status 404 returned error can't find the container with id 46ba2c47bd37afc3e79c3ce7775c23577925d2ef1ea0306dd4aef713e707149b
Sep 17 15:50:16.089770 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:16.089683    2574 generic.go:356] "Generic (PLEG): container finished" podID="33f0b0fb-64fa-4e36-943a-07efcc62da86" containerID="a26f9ee201f6e9060f1aee954f402528acd12c0bb5beca59bdef7a872b0f55c8" exitCode=0
Sep 17 15:50:16.089770 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:16.089723    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6"
Sep 17 15:50:16.089955 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:16.089773    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6" event={"ID":"33f0b0fb-64fa-4e36-943a-07efcc62da86","Type":"ContainerDied","Data":"a26f9ee201f6e9060f1aee954f402528acd12c0bb5beca59bdef7a872b0f55c8"}
Sep 17 15:50:16.089955 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:16.089818    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-74b7f9c78b-6hwz6" event={"ID":"33f0b0fb-64fa-4e36-943a-07efcc62da86","Type":"ContainerDied","Data":"76a4cf25bca892713ca0a4ad9f48c288fe55ca2f74974ed71ac506105bb9a8f2"}
Sep 17 15:50:16.089955 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:16.089844    2574 scope.go:117] "RemoveContainer" containerID="a26f9ee201f6e9060f1aee954f402528acd12c0bb5beca59bdef7a872b0f55c8"
Sep 17 15:50:16.091175 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:16.091148    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-74b7f9c78b-78vlf" event={"ID":"299537d9-a2d7-4dd2-ae91-1e72e66b69f7","Type":"ContainerStarted","Data":"46ba2c47bd37afc3e79c3ce7775c23577925d2ef1ea0306dd4aef713e707149b"}
Sep 17 15:50:16.098421 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:16.098400    2574 scope.go:117] "RemoveContainer" containerID="a26f9ee201f6e9060f1aee954f402528acd12c0bb5beca59bdef7a872b0f55c8"
Sep 17 15:50:16.098715 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:50:16.098694    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a26f9ee201f6e9060f1aee954f402528acd12c0bb5beca59bdef7a872b0f55c8\": container with ID starting with a26f9ee201f6e9060f1aee954f402528acd12c0bb5beca59bdef7a872b0f55c8 not found: ID does not exist" containerID="a26f9ee201f6e9060f1aee954f402528acd12c0bb5beca59bdef7a872b0f55c8"
Sep 17 15:50:16.098788 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:16.098727    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26f9ee201f6e9060f1aee954f402528acd12c0bb5beca59bdef7a872b0f55c8"} err="failed to get container status \"a26f9ee201f6e9060f1aee954f402528acd12c0bb5beca59bdef7a872b0f55c8\": rpc error: code = NotFound desc = could not find container \"a26f9ee201f6e9060f1aee954f402528acd12c0bb5beca59bdef7a872b0f55c8\": container with ID starting with a26f9ee201f6e9060f1aee954f402528acd12c0bb5beca59bdef7a872b0f55c8 not found: ID does not exist"
Sep 17 15:50:16.111532 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:16.111344    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-74b7f9c78b-6hwz6"]
Sep 17 15:50:16.113277 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:16.113260    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-74b7f9c78b-6hwz6"]
Sep 17 15:50:17.049324 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:17.049287    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33f0b0fb-64fa-4e36-943a-07efcc62da86" path="/var/lib/kubelet/pods/33f0b0fb-64fa-4e36-943a-07efcc62da86/volumes"
Sep 17 15:50:17.098140 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:17.098103    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-74b7f9c78b-78vlf" event={"ID":"299537d9-a2d7-4dd2-ae91-1e72e66b69f7","Type":"ContainerStarted","Data":"baff5c13f142ee8193062ea8a07d641cf10e588b6cd7a438c44c1bb3cdd02de3"}
Sep 17 15:50:17.098298 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:17.098246    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-74b7f9c78b-78vlf"
Sep 17 15:50:17.116904 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:17.116853    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-74b7f9c78b-78vlf" podStartSLOduration=1.715801181 podStartE2EDuration="2.116838622s" totalImagesPullingTime="401.037441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:17.11470524 +0000 UTC m=+682.586545085" watchObservedRunningTime="2026-09-17 15:50:17.116838622 +0000 UTC m=+682.588678468"
Sep 17 15:50:26.763836 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:26.763789    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:28.382675 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:28.382613    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:48.108402 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:48.108317    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-74b7f9c78b-78vlf"
Sep 17 15:50:49.336912 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.336868    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:49.391996 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.391958    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-47xz5"]
Sep 17 15:50:49.392425 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.392406    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33f0b0fb-64fa-4e36-943a-07efcc62da86" containerName="manager"
Sep 17 15:50:49.392531 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.392428    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="33f0b0fb-64fa-4e36-943a-07efcc62da86" containerName="manager"
Sep 17 15:50:49.392590 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.392566    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="33f0b0fb-64fa-4e36-943a-07efcc62da86" containerName="manager"
Sep 17 15:50:49.394987 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.394964    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-5x6jm"]
Sep 17 15:50:49.395105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.395069    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-47xz5"
Sep 17 15:50:49.397508 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.397482    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-b9fcd\""
Sep 17 15:50:49.397508 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.397503    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-5x6jm"
Sep 17 15:50:49.397508 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.397434    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 15:50:49.399561 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.399538    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-8gxxj\""
Sep 17 15:50:49.399661 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.399612    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 15:50:49.399735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.399692    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 15:50:49.404888 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.404864    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-47xz5"]
Sep 17 15:50:49.407953 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.407932    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-5x6jm"]
Sep 17 15:50:49.430271 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.430236    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:49.495670 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.495627    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vxf7\" (UniqueName: \"kubernetes.io/projected/e281b834-ad84-4766-be56-4a9626f6a1e1-kube-api-access-8vxf7\") pod \"model-serving-api-7b954ffc95-47xz5\" (UID: \"e281b834-ad84-4766-be56-4a9626f6a1e1\") " pod="kserve/model-serving-api-7b954ffc95-47xz5"
Sep 17 15:50:49.495670 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.495677    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/135c2239-af06-4865-ada4-4414462f4965-cert\") pod \"odh-model-controller-7b966d7984-5x6jm\" (UID: \"135c2239-af06-4865-ada4-4414462f4965\") " pod="kserve/odh-model-controller-7b966d7984-5x6jm"
Sep 17 15:50:49.495884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.495733    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e281b834-ad84-4766-be56-4a9626f6a1e1-tls-certs\") pod \"model-serving-api-7b954ffc95-47xz5\" (UID: \"e281b834-ad84-4766-be56-4a9626f6a1e1\") " pod="kserve/model-serving-api-7b954ffc95-47xz5"
Sep 17 15:50:49.495884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.495789    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/135c2239-af06-4865-ada4-4414462f4965-metrics-certs\") pod \"odh-model-controller-7b966d7984-5x6jm\" (UID: \"135c2239-af06-4865-ada4-4414462f4965\") " pod="kserve/odh-model-controller-7b966d7984-5x6jm"
Sep 17 15:50:49.495884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.495829    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zslqg\" (UniqueName: \"kubernetes.io/projected/135c2239-af06-4865-ada4-4414462f4965-kube-api-access-zslqg\") pod \"odh-model-controller-7b966d7984-5x6jm\" (UID: \"135c2239-af06-4865-ada4-4414462f4965\") " pod="kserve/odh-model-controller-7b966d7984-5x6jm"
Sep 17 15:50:49.566003 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.565950    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:49.597072 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.596972    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vxf7\" (UniqueName: \"kubernetes.io/projected/e281b834-ad84-4766-be56-4a9626f6a1e1-kube-api-access-8vxf7\") pod \"model-serving-api-7b954ffc95-47xz5\" (UID: \"e281b834-ad84-4766-be56-4a9626f6a1e1\") " pod="kserve/model-serving-api-7b954ffc95-47xz5"
Sep 17 15:50:49.597072 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.597015    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/135c2239-af06-4865-ada4-4414462f4965-cert\") pod \"odh-model-controller-7b966d7984-5x6jm\" (UID: \"135c2239-af06-4865-ada4-4414462f4965\") " pod="kserve/odh-model-controller-7b966d7984-5x6jm"
Sep 17 15:50:49.597072 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.597054    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e281b834-ad84-4766-be56-4a9626f6a1e1-tls-certs\") pod \"model-serving-api-7b954ffc95-47xz5\" (UID: \"e281b834-ad84-4766-be56-4a9626f6a1e1\") " pod="kserve/model-serving-api-7b954ffc95-47xz5"
Sep 17 15:50:49.597377 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.597088    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/135c2239-af06-4865-ada4-4414462f4965-metrics-certs\") pod \"odh-model-controller-7b966d7984-5x6jm\" (UID: \"135c2239-af06-4865-ada4-4414462f4965\") " pod="kserve/odh-model-controller-7b966d7984-5x6jm"
Sep 17 15:50:49.597377 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.597133    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zslqg\" (UniqueName: \"kubernetes.io/projected/135c2239-af06-4865-ada4-4414462f4965-kube-api-access-zslqg\") pod \"odh-model-controller-7b966d7984-5x6jm\" (UID: \"135c2239-af06-4865-ada4-4414462f4965\") " pod="kserve/odh-model-controller-7b966d7984-5x6jm"
Sep 17 15:50:49.597377 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:50:49.597359    2574 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 17 15:50:49.597581 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:50:49.597474    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/135c2239-af06-4865-ada4-4414462f4965-metrics-certs podName:135c2239-af06-4865-ada4-4414462f4965 nodeName:}" failed. No retries permitted until 2026-09-17 15:50:50.097412944 +0000 UTC m=+715.569252771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/135c2239-af06-4865-ada4-4414462f4965-metrics-certs") pod "odh-model-controller-7b966d7984-5x6jm" (UID: "135c2239-af06-4865-ada4-4414462f4965") : secret "odh-model-controller-metrics-tls" not found
Sep 17 15:50:49.599715 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.599688    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e281b834-ad84-4766-be56-4a9626f6a1e1-tls-certs\") pod \"model-serving-api-7b954ffc95-47xz5\" (UID: \"e281b834-ad84-4766-be56-4a9626f6a1e1\") " pod="kserve/model-serving-api-7b954ffc95-47xz5"
Sep 17 15:50:49.599821 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.599778    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/135c2239-af06-4865-ada4-4414462f4965-cert\") pod \"odh-model-controller-7b966d7984-5x6jm\" (UID: \"135c2239-af06-4865-ada4-4414462f4965\") " pod="kserve/odh-model-controller-7b966d7984-5x6jm"
Sep 17 15:50:49.607024 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.607002    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zslqg\" (UniqueName: \"kubernetes.io/projected/135c2239-af06-4865-ada4-4414462f4965-kube-api-access-zslqg\") pod \"odh-model-controller-7b966d7984-5x6jm\" (UID: \"135c2239-af06-4865-ada4-4414462f4965\") " pod="kserve/odh-model-controller-7b966d7984-5x6jm"
Sep 17 15:50:49.607164 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.607032    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vxf7\" (UniqueName: \"kubernetes.io/projected/e281b834-ad84-4766-be56-4a9626f6a1e1-kube-api-access-8vxf7\") pod \"model-serving-api-7b954ffc95-47xz5\" (UID: \"e281b834-ad84-4766-be56-4a9626f6a1e1\") " pod="kserve/model-serving-api-7b954ffc95-47xz5"
Sep 17 15:50:49.666926 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.666882    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:49.709570 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.709538    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-47xz5"
Sep 17 15:50:49.981571 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.981534    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-47xz5"]
Sep 17 15:50:49.985303 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:50:49.985274    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode281b834_ad84_4766_be56_4a9626f6a1e1.slice/crio-4a55578d87db2971b59905bbf8b69e63f730f47f71ac86f9051c5a842dce8c60 WatchSource:0}: Error finding container 4a55578d87db2971b59905bbf8b69e63f730f47f71ac86f9051c5a842dce8c60: Status 404 returned error can't find the container with id 4a55578d87db2971b59905bbf8b69e63f730f47f71ac86f9051c5a842dce8c60
Sep 17 15:50:49.987750 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:49.987727    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:50:50.066347 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:50.066312    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:50.101378 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:50.101347    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/135c2239-af06-4865-ada4-4414462f4965-metrics-certs\") pod \"odh-model-controller-7b966d7984-5x6jm\" (UID: \"135c2239-af06-4865-ada4-4414462f4965\") " pod="kserve/odh-model-controller-7b966d7984-5x6jm"
Sep 17 15:50:50.103871 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:50.103852    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/135c2239-af06-4865-ada4-4414462f4965-metrics-certs\") pod \"odh-model-controller-7b966d7984-5x6jm\" (UID: \"135c2239-af06-4865-ada4-4414462f4965\") " pod="kserve/odh-model-controller-7b966d7984-5x6jm"
Sep 17 15:50:50.255587 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:50.255501    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-47xz5" event={"ID":"e281b834-ad84-4766-be56-4a9626f6a1e1","Type":"ContainerStarted","Data":"4a55578d87db2971b59905bbf8b69e63f730f47f71ac86f9051c5a842dce8c60"}
Sep 17 15:50:50.317970 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:50.317931    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-5x6jm"
Sep 17 15:50:50.592957 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:50.592775    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-5x6jm"]
Sep 17 15:50:50.596717 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:50:50.596680    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135c2239_af06_4865_ada4_4414462f4965.slice/crio-134693176f779c1859859a81105e66e3d355af59935177a872e6712f6bd58bbe WatchSource:0}: Error finding container 134693176f779c1859859a81105e66e3d355af59935177a872e6712f6bd58bbe: Status 404 returned error can't find the container with id 134693176f779c1859859a81105e66e3d355af59935177a872e6712f6bd58bbe
Sep 17 15:50:50.979875 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:50.979839    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:51.033111 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:51.033051    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:51.265215 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:51.265116    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-5x6jm" event={"ID":"135c2239-af06-4865-ada4-4414462f4965","Type":"ContainerStarted","Data":"134693176f779c1859859a81105e66e3d355af59935177a872e6712f6bd58bbe"}
Sep 17 15:50:54.289948 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:54.289908    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-5x6jm" event={"ID":"135c2239-af06-4865-ada4-4414462f4965","Type":"ContainerStarted","Data":"96109077c2548de00d6b8d7210428757dc4e1731a62f3d06f222a0a91015a35b"}
Sep 17 15:50:54.290434 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:54.290091    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-5x6jm"
Sep 17 15:50:54.291469 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:54.291432    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-47xz5" event={"ID":"e281b834-ad84-4766-be56-4a9626f6a1e1","Type":"ContainerStarted","Data":"f288ed7608e71a6210ceb3e13d596ab82097fdef4137644ed7e8d4e929fc9618"}
Sep 17 15:50:54.291517 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:54.291491    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-47xz5"
Sep 17 15:50:54.309250 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:54.309193    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-5x6jm" podStartSLOduration=2.3353314 podStartE2EDuration="5.309177261s" totalImagesPullingTime="2.973845861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:54.307532121 +0000 UTC m=+719.779371973" watchObservedRunningTime="2026-09-17 15:50:54.309177261 +0000 UTC m=+719.781017144"
Sep 17 15:50:54.332705 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:50:54.332641    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-47xz5" podStartSLOduration=1.757589064 podStartE2EDuration="5.332619454s" totalImagesPullingTime="3.57503039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:54.331918982 +0000 UTC m=+719.803758832" watchObservedRunningTime="2026-09-17 15:50:54.332619454 +0000 UTC m=+719.804459301"
Sep 17 15:51:01.347117 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:01.347071    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:51:01.447017 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:01.446982    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:51:05.298094 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:05.298063    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-5x6jm"
Sep 17 15:51:05.300146 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:05.300122    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-47xz5"
Sep 17 15:51:06.188645 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:06.188608    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-qhnrx"]
Sep 17 15:51:06.191956 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:06.191939    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qhnrx"
Sep 17 15:51:06.194122 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:06.194099    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 15:51:06.201362 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:06.201338    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qhnrx"]
Sep 17 15:51:06.353281 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:06.353252    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfpp7\" (UniqueName: \"kubernetes.io/projected/10dadd7b-9637-45d1-b83a-c2d8180cf080-kube-api-access-lfpp7\") pod \"s3-init-qhnrx\" (UID: \"10dadd7b-9637-45d1-b83a-c2d8180cf080\") " pod="kserve/s3-init-qhnrx"
Sep 17 15:51:06.353708 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:06.353306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dadd7b-9637-45d1-b83a-c2d8180cf080-model-cache\") pod \"s3-init-qhnrx\" (UID: \"10dadd7b-9637-45d1-b83a-c2d8180cf080\") " pod="kserve/s3-init-qhnrx"
Sep 17 15:51:06.454227 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:06.454129    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dadd7b-9637-45d1-b83a-c2d8180cf080-model-cache\") pod \"s3-init-qhnrx\" (UID: \"10dadd7b-9637-45d1-b83a-c2d8180cf080\") " pod="kserve/s3-init-qhnrx"
Sep 17 15:51:06.454227 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:06.454225    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfpp7\" (UniqueName: \"kubernetes.io/projected/10dadd7b-9637-45d1-b83a-c2d8180cf080-kube-api-access-lfpp7\") pod \"s3-init-qhnrx\" (UID: \"10dadd7b-9637-45d1-b83a-c2d8180cf080\") " pod="kserve/s3-init-qhnrx"
Sep 17 15:51:06.454593 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:06.454570    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dadd7b-9637-45d1-b83a-c2d8180cf080-model-cache\") pod \"s3-init-qhnrx\" (UID: \"10dadd7b-9637-45d1-b83a-c2d8180cf080\") " pod="kserve/s3-init-qhnrx"
Sep 17 15:51:06.462537 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:06.462508    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfpp7\" (UniqueName: \"kubernetes.io/projected/10dadd7b-9637-45d1-b83a-c2d8180cf080-kube-api-access-lfpp7\") pod \"s3-init-qhnrx\" (UID: \"10dadd7b-9637-45d1-b83a-c2d8180cf080\") " pod="kserve/s3-init-qhnrx"
Sep 17 15:51:06.502297 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:06.502256    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qhnrx"
Sep 17 15:51:06.779468 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:06.779286    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qhnrx"]
Sep 17 15:51:06.782123 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:51:06.782092    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10dadd7b_9637_45d1_b83a_c2d8180cf080.slice/crio-f8c85eb70544787e2847b951a93f4c574de219ee101508bb16a09d8898668b03 WatchSource:0}: Error finding container f8c85eb70544787e2847b951a93f4c574de219ee101508bb16a09d8898668b03: Status 404 returned error can't find the container with id f8c85eb70544787e2847b951a93f4c574de219ee101508bb16a09d8898668b03
Sep 17 15:51:07.354565 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:07.354530    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qhnrx" event={"ID":"10dadd7b-9637-45d1-b83a-c2d8180cf080","Type":"ContainerStarted","Data":"f8c85eb70544787e2847b951a93f4c574de219ee101508bb16a09d8898668b03"}
Sep 17 15:51:10.372807 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:10.372767    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qhnrx" event={"ID":"10dadd7b-9637-45d1-b83a-c2d8180cf080","Type":"ContainerStarted","Data":"dff033d9e97848f6f98c6db37c430492803e8fbb8492f2d5b16af9507a3e727c"}
Sep 17 15:51:13.388334 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:13.388298    2574 generic.go:356] "Generic (PLEG): container finished" podID="10dadd7b-9637-45d1-b83a-c2d8180cf080" containerID="dff033d9e97848f6f98c6db37c430492803e8fbb8492f2d5b16af9507a3e727c" exitCode=0
Sep 17 15:51:13.388736 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:13.388378    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qhnrx" event={"ID":"10dadd7b-9637-45d1-b83a-c2d8180cf080","Type":"ContainerDied","Data":"dff033d9e97848f6f98c6db37c430492803e8fbb8492f2d5b16af9507a3e727c"}
Sep 17 15:51:18.418380 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:18.418341    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qhnrx" event={"ID":"10dadd7b-9637-45d1-b83a-c2d8180cf080","Type":"ContainerStarted","Data":"fa466987c9e99e047e25bda01ef821067ac98e8c0835289af8e192c83664006b"}
Sep 17 15:51:18.438615 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:51:18.438561    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-qhnrx" podStartSLOduration=2.202056751 podStartE2EDuration="12.438545393s" totalImagesPullingTime="7.575628739s" totalInitContainerRuntime="2.660859903s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:51:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:51:18.43658794 +0000 UTC m=+743.908427788" watchObservedRunningTime="2026-09-17 15:51:18.438545393 +0000 UTC m=+743.910385238"
Sep 17 15:52:23.730404 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:23.730366    2574 generic.go:356] "Generic (PLEG): container finished" podID="10dadd7b-9637-45d1-b83a-c2d8180cf080" containerID="fa466987c9e99e047e25bda01ef821067ac98e8c0835289af8e192c83664006b" exitCode=0
Sep 17 15:52:23.730826 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:23.730454    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qhnrx" event={"ID":"10dadd7b-9637-45d1-b83a-c2d8180cf080","Type":"ContainerDied","Data":"fa466987c9e99e047e25bda01ef821067ac98e8c0835289af8e192c83664006b"}
Sep 17 15:52:24.864757 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:24.864726    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qhnrx"
Sep 17 15:52:24.951866 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:24.951823    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dadd7b-9637-45d1-b83a-c2d8180cf080-model-cache\") pod \"10dadd7b-9637-45d1-b83a-c2d8180cf080\" (UID: \"10dadd7b-9637-45d1-b83a-c2d8180cf080\") "
Sep 17 15:52:24.952047 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:24.951960    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfpp7\" (UniqueName: \"kubernetes.io/projected/10dadd7b-9637-45d1-b83a-c2d8180cf080-kube-api-access-lfpp7\") pod \"10dadd7b-9637-45d1-b83a-c2d8180cf080\" (UID: \"10dadd7b-9637-45d1-b83a-c2d8180cf080\") "
Sep 17 15:52:24.954081 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:24.954046    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10dadd7b-9637-45d1-b83a-c2d8180cf080-kube-api-access-lfpp7" (OuterVolumeSpecName: "kube-api-access-lfpp7") pod "10dadd7b-9637-45d1-b83a-c2d8180cf080" (UID: "10dadd7b-9637-45d1-b83a-c2d8180cf080"). InnerVolumeSpecName "kube-api-access-lfpp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:52:24.970768 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:24.970732    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dadd7b-9637-45d1-b83a-c2d8180cf080-model-cache" (OuterVolumeSpecName: "model-cache") pod "10dadd7b-9637-45d1-b83a-c2d8180cf080" (UID: "10dadd7b-9637-45d1-b83a-c2d8180cf080"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:52:25.052798 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:25.052701    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfpp7\" (UniqueName: \"kubernetes.io/projected/10dadd7b-9637-45d1-b83a-c2d8180cf080-kube-api-access-lfpp7\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:52:25.052798 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:25.052729    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dadd7b-9637-45d1-b83a-c2d8180cf080-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:52:25.741117 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:25.741084    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qhnrx" event={"ID":"10dadd7b-9637-45d1-b83a-c2d8180cf080","Type":"ContainerDied","Data":"f8c85eb70544787e2847b951a93f4c574de219ee101508bb16a09d8898668b03"}
Sep 17 15:52:25.741117 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:25.741123    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8c85eb70544787e2847b951a93f4c574de219ee101508bb16a09d8898668b03"
Sep 17 15:52:25.741330 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:25.741124    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qhnrx"
Sep 17 15:52:26.130008 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:26.129968    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:52:26.223213 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:26.223179    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:52:38.519652 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:38.519603    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:52:38.719435 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:38.719381    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:52:44.406754 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.406713    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"]
Sep 17 15:52:44.407313 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.407291    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10dadd7b-9637-45d1-b83a-c2d8180cf080" containerName="s3-init"
Sep 17 15:52:44.407401 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.407315    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="10dadd7b-9637-45d1-b83a-c2d8180cf080" containerName="s3-init"
Sep 17 15:52:44.407401 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.407333    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10dadd7b-9637-45d1-b83a-c2d8180cf080" containerName="download-hf-model"
Sep 17 15:52:44.407401 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.407341    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="10dadd7b-9637-45d1-b83a-c2d8180cf080" containerName="download-hf-model"
Sep 17 15:52:44.407566 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.407490    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="10dadd7b-9637-45d1-b83a-c2d8180cf080" containerName="s3-init"
Sep 17 15:52:44.410584 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.410561    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.411055 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.411016    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:52:44.413645 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.413623    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-qvrhc\""
Sep 17 15:52:44.413813 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.413676    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 15:52:44.413874 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.413837    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 15:52:44.414135 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.414115    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 15:52:44.424628 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.424603    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"]
Sep 17 15:52:44.526119 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.526067    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-home\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.526119 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.526128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.526381 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.526183    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.526381 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.526254    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v966\" (UniqueName: \"kubernetes.io/projected/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-kube-api-access-9v966\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.526381 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.526281    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.526381 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.526340    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-dshm\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.627630 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.627588    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.627800 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.627670    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v966\" (UniqueName: \"kubernetes.io/projected/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-kube-api-access-9v966\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.627800 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.627699    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.627800 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.627731    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-dshm\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.627800 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.627780    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-home\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.627972 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.627817    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.628155 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.628125    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.628259 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.628182    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-home\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.628308 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.628256    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.630013 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.629989    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-dshm\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.630180 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.630165    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.638504 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.638477    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v966\" (UniqueName: \"kubernetes.io/projected/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-kube-api-access-9v966\") pod \"fc-smoke-test-kserve-7f4687897-7xbhz\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.721958 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.721877    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:44.845473 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.845428    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"]
Sep 17 15:52:44.848990 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.848974    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"
Sep 17 15:52:44.851713 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.851687    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-zhg64\""
Sep 17 15:52:44.863732 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.863705    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"]
Sep 17 15:52:44.899395 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.899365    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:52:44.930526 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.930488    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d2316b-e4a3-48a5-905e-7ce7c66c5430-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt\" (UID: \"24d2316b-e4a3-48a5-905e-7ce7c66c5430\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"
Sep 17 15:52:44.930681 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.930565    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2nhc\" (UniqueName: \"kubernetes.io/projected/24d2316b-e4a3-48a5-905e-7ce7c66c5430-kube-api-access-c2nhc\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt\" (UID: \"24d2316b-e4a3-48a5-905e-7ce7c66c5430\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"
Sep 17 15:52:44.999227 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:44.999196    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"]
Sep 17 15:52:45.000986 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:52:45.000960    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ba0ec0_d6a8_4732_9ae0_8d82e9eaf7f5.slice/crio-94f0463ae0d7bc5c3551f1e527ec7d10d4916e6737025f299acdee4c15b30eab WatchSource:0}: Error finding container 94f0463ae0d7bc5c3551f1e527ec7d10d4916e6737025f299acdee4c15b30eab: Status 404 returned error can't find the container with id 94f0463ae0d7bc5c3551f1e527ec7d10d4916e6737025f299acdee4c15b30eab
Sep 17 15:52:45.031327 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:45.031297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d2316b-e4a3-48a5-905e-7ce7c66c5430-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt\" (UID: \"24d2316b-e4a3-48a5-905e-7ce7c66c5430\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"
Sep 17 15:52:45.031474 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:45.031344    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2nhc\" (UniqueName: \"kubernetes.io/projected/24d2316b-e4a3-48a5-905e-7ce7c66c5430-kube-api-access-c2nhc\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt\" (UID: \"24d2316b-e4a3-48a5-905e-7ce7c66c5430\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"
Sep 17 15:52:45.034079 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:45.034058    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d2316b-e4a3-48a5-905e-7ce7c66c5430-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt\" (UID: \"24d2316b-e4a3-48a5-905e-7ce7c66c5430\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"
Sep 17 15:52:45.044826 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:45.044800    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2nhc\" (UniqueName: \"kubernetes.io/projected/24d2316b-e4a3-48a5-905e-7ce7c66c5430-kube-api-access-c2nhc\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt\" (UID: \"24d2316b-e4a3-48a5-905e-7ce7c66c5430\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"
Sep 17 15:52:45.159742 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:45.159712    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"
Sep 17 15:52:45.444710 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:45.444622    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"]
Sep 17 15:52:45.514341 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:52:45.514300    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d2316b_e4a3_48a5_905e_7ce7c66c5430.slice/crio-87250448c3f317d3d0dbc0b6e9493e120cbbe693b502d297f1157194416e75a4 WatchSource:0}: Error finding container 87250448c3f317d3d0dbc0b6e9493e120cbbe693b502d297f1157194416e75a4: Status 404 returned error can't find the container with id 87250448c3f317d3d0dbc0b6e9493e120cbbe693b502d297f1157194416e75a4
Sep 17 15:52:45.847652 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:45.847551    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz" event={"ID":"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5","Type":"ContainerStarted","Data":"94f0463ae0d7bc5c3551f1e527ec7d10d4916e6737025f299acdee4c15b30eab"}
Sep 17 15:52:45.849633 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:45.849570    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt" event={"ID":"24d2316b-e4a3-48a5-905e-7ce7c66c5430","Type":"ContainerStarted","Data":"87250448c3f317d3d0dbc0b6e9493e120cbbe693b502d297f1157194416e75a4"}
Sep 17 15:52:46.857702 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:46.857658    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz" event={"ID":"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5","Type":"ContainerStarted","Data":"8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94"}
Sep 17 15:52:46.883721 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:46.883658    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz" podStartSLOduration=1.1616252089999999 podStartE2EDuration="2.883640264s" totalImagesPullingTime="1.722015055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:52:46.881788406 +0000 UTC m=+832.353628253" watchObservedRunningTime="2026-09-17 15:52:46.883640264 +0000 UTC m=+832.355480111"
Sep 17 15:52:47.722242 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:47.722203    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:48.871783 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:48.871733    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt" event={"ID":"24d2316b-e4a3-48a5-905e-7ce7c66c5430","Type":"ContainerStarted","Data":"a666a2be8001bb80fe0be98f9c7eda268eb12f285cd63ffdd93d7f9c46543655"}
Sep 17 15:52:48.915573 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:48.915517    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt" podStartSLOduration=2.399213846 podStartE2EDuration="4.915500122s" totalImagesPullingTime="2.516286276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:52:48.913958375 +0000 UTC m=+834.385798221" watchObservedRunningTime="2026-09-17 15:52:48.915500122 +0000 UTC m=+834.387339967"
Sep 17 15:52:49.876929 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:49.876899    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"
Sep 17 15:52:54.722264 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:54.722218    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:54.735151 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:54.735122    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:52:54.911581 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:52:54.911507    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:53:03.944902 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:03.944851    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:53:04.258222 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.258052    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"]
Sep 17 15:53:04.262551 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.262526    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"
Sep 17 15:53:04.270499 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.270421    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 15:53:04.270499 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.270504    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-7wj8f\""
Sep 17 15:53:04.270895 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.270859    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 15:53:04.271138 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.271120    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-wkvd8\""
Sep 17 15:53:04.271235 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.271219    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 15:53:04.287929 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.287899    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"]
Sep 17 15:53:04.300568 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.300534    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:53:04.309303 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.309269    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d5e532-2f4a-4165-82a0-630ac487ca87-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m\" (UID: \"a1d5e532-2f4a-4165-82a0-630ac487ca87\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"
Sep 17 15:53:04.309519 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.309330    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndc8x\" (UniqueName: \"kubernetes.io/projected/a1d5e532-2f4a-4165-82a0-630ac487ca87-kube-api-access-ndc8x\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m\" (UID: \"a1d5e532-2f4a-4165-82a0-630ac487ca87\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"
Sep 17 15:53:04.410758 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.410716    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d5e532-2f4a-4165-82a0-630ac487ca87-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m\" (UID: \"a1d5e532-2f4a-4165-82a0-630ac487ca87\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"
Sep 17 15:53:04.410758 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.410765    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndc8x\" (UniqueName: \"kubernetes.io/projected/a1d5e532-2f4a-4165-82a0-630ac487ca87-kube-api-access-ndc8x\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m\" (UID: \"a1d5e532-2f4a-4165-82a0-630ac487ca87\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"
Sep 17 15:53:04.413351 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.413321    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d5e532-2f4a-4165-82a0-630ac487ca87-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m\" (UID: \"a1d5e532-2f4a-4165-82a0-630ac487ca87\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"
Sep 17 15:53:04.435830 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.435795    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndc8x\" (UniqueName: \"kubernetes.io/projected/a1d5e532-2f4a-4165-82a0-630ac487ca87-kube-api-access-ndc8x\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m\" (UID: \"a1d5e532-2f4a-4165-82a0-630ac487ca87\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"
Sep 17 15:53:04.573735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.573640    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"
Sep 17 15:53:04.862928 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.862897    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"]
Sep 17 15:53:04.865790 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:53:04.865764    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d5e532_2f4a_4165_82a0_630ac487ca87.slice/crio-b3ebe9e18bedc213b0bb5d1f6faac6c2eca9c540f91e9d72884e9ebf20dc4667 WatchSource:0}: Error finding container b3ebe9e18bedc213b0bb5d1f6faac6c2eca9c540f91e9d72884e9ebf20dc4667: Status 404 returned error can't find the container with id b3ebe9e18bedc213b0bb5d1f6faac6c2eca9c540f91e9d72884e9ebf20dc4667
Sep 17 15:53:04.952272 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.952229    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m" event={"ID":"a1d5e532-2f4a-4165-82a0-630ac487ca87","Type":"ContainerStarted","Data":"afd57fc52463abed1eb7a66b6cbc62765fe90df9dbadd128ef4254da9c46edc8"}
Sep 17 15:53:04.952768 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.952292    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m" event={"ID":"a1d5e532-2f4a-4165-82a0-630ac487ca87","Type":"ContainerStarted","Data":"b3ebe9e18bedc213b0bb5d1f6faac6c2eca9c540f91e9d72884e9ebf20dc4667"}
Sep 17 15:53:04.952768 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.952356    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"
Sep 17 15:53:04.992012 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:04.991958    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m" podStartSLOduration=0.991938469 podStartE2EDuration="991.938469ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:53:04.990470743 +0000 UTC m=+850.462310582" watchObservedRunningTime="2026-09-17 15:53:04.991938469 +0000 UTC m=+850.463778315"
Sep 17 15:53:14.589910 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:14.589861    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:53:20.883591 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:20.883561    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"
Sep 17 15:53:25.005842 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:25.005793    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:53:25.110420 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:25.110381    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:53:35.958613 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:35.958582    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"
Sep 17 15:53:40.699112 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:40.699080    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:53:45.793769 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:45.793733    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"]
Sep 17 15:53:45.856332 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:45.856296    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:53:45.958965 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:45.958927    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m" podUID="a1d5e532-2f4a-4165-82a0-630ac487ca87" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:53:46.003047 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:46.003008    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:53:49.015735 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.015644    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"]
Sep 17 15:53:49.034195 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.034153    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"]
Sep 17 15:53:49.048393 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:53:49.048344    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 15:53:49.048393 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:53:49.048393 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 15:53:49.048393 ip-10-0-129-37 kubenswrapper[2574]:  > containerID="8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94" cmd=["/bin/sleep","15"]
Sep 17 15:53:49.048769 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.048419    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz" message="" err=<
Sep 17 15:53:49.048769 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:53:49.048769 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 15:53:49.048769 ip-10-0-129-37 kubenswrapper[2574]:  >
Sep 17 15:53:49.048769 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:53:49.048435    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 15:53:49.048769 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:53:49.048769 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 15:53:49.048769 ip-10-0-129-37 kubenswrapper[2574]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz" podUID="e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5" containerName="main" containerID="cri-o://8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94"
Sep 17 15:53:49.048769 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.048586    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz" podUID="e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5" containerName="main" containerID="cri-o://8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94" gracePeriod=60
Sep 17 15:53:49.072986 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.072943    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:53:49.240903 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.240850    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:53:49.404015 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.403988    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:53:49.523654 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.523622    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-home\") pod \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") "
Sep 17 15:53:49.523837 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.523684    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v966\" (UniqueName: \"kubernetes.io/projected/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-kube-api-access-9v966\") pod \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") "
Sep 17 15:53:49.523837 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.523710    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-tls-certs\") pod \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") "
Sep 17 15:53:49.523837 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.523734    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-tmp-dir\") pod \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") "
Sep 17 15:53:49.523837 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.523751    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-model-cache\") pod \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") "
Sep 17 15:53:49.524062 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.523841    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-dshm\") pod \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\" (UID: \"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5\") "
Sep 17 15:53:49.524062 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.523913    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-home" (OuterVolumeSpecName: "home") pod "e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5" (UID: "e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:53:49.524062 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.523996    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5" (UID: "e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:53:49.524203 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.524083    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5" (UID: "e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:53:49.524240 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.524196    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:53:49.524240 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.524224    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:53:49.524309 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.524241    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:53:49.526084 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.526061    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-dshm" (OuterVolumeSpecName: "dshm") pod "e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5" (UID: "e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:53:49.526187 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.526122    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5" (UID: "e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:53:49.526187 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.526155    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-kube-api-access-9v966" (OuterVolumeSpecName: "kube-api-access-9v966") pod "e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5" (UID: "e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5"). InnerVolumeSpecName "kube-api-access-9v966". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:53:49.624711 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.624674    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:53:49.624711 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.624711    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v966\" (UniqueName: \"kubernetes.io/projected/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-kube-api-access-9v966\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:53:49.624711 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:49.624722    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:53:50.171865 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:50.171829    2574 generic.go:356] "Generic (PLEG): container finished" podID="e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5" containerID="8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94" exitCode=0
Sep 17 15:53:50.172237 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:50.171878    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"
Sep 17 15:53:50.172237 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:50.171918    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz" event={"ID":"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5","Type":"ContainerDied","Data":"8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94"}
Sep 17 15:53:50.172237 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:50.171986    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz" event={"ID":"e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5","Type":"ContainerDied","Data":"94f0463ae0d7bc5c3551f1e527ec7d10d4916e6737025f299acdee4c15b30eab"}
Sep 17 15:53:50.172237 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:50.172010    2574 scope.go:117] "RemoveContainer" containerID="8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94"
Sep 17 15:53:50.181670 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:50.181650    2574 scope.go:117] "RemoveContainer" containerID="8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94"
Sep 17 15:53:50.181957 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:53:50.181937    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94\": container with ID starting with 8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94 not found: ID does not exist" containerID="8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94"
Sep 17 15:53:50.182005 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:50.181967    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94"} err="failed to get container status \"8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94\": rpc error: code = NotFound desc = could not find container \"8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94\": container with ID starting with 8dd705a9218c364bbbc609812c28d48e770ff420111644aaf2c69ab5b60ecb94 not found: ID does not exist"
Sep 17 15:53:50.194907 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:50.194878    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"]
Sep 17 15:53:50.199754 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:50.199729    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7xbhz"]
Sep 17 15:53:50.883140 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:50.883101    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt" podUID="24d2316b-e4a3-48a5-905e-7ce7c66c5430" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:53:51.049082 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:51.049041    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5" path="/var/lib/kubelet/pods/e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5/volumes"
Sep 17 15:53:55.959165 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:55.959124    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m" podUID="a1d5e532-2f4a-4165-82a0-630ac487ca87" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:53:57.300342 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:57.300299    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 17 15:53:57.300342 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:53:57.300335    2574 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m" podUID="a1d5e532-2f4a-4165-82a0-630ac487ca87" containerName="main" containerID="cri-o://afd57fc52463abed1eb7a66b6cbc62765fe90df9dbadd128ef4254da9c46edc8"
Sep 17 15:53:57.300633 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:57.300376    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m" podUID="a1d5e532-2f4a-4165-82a0-630ac487ca87" containerName="main" containerID="cri-o://afd57fc52463abed1eb7a66b6cbc62765fe90df9dbadd128ef4254da9c46edc8" gracePeriod=49
Sep 17 15:53:57.452502 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:57.452470    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m_a1d5e532-2f4a-4165-82a0-630ac487ca87/main/0.log"
Sep 17 15:53:57.454809 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:57.454791    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"
Sep 17 15:53:57.595069 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:57.594963    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndc8x\" (UniqueName: \"kubernetes.io/projected/a1d5e532-2f4a-4165-82a0-630ac487ca87-kube-api-access-ndc8x\") pod \"a1d5e532-2f4a-4165-82a0-630ac487ca87\" (UID: \"a1d5e532-2f4a-4165-82a0-630ac487ca87\") "
Sep 17 15:53:57.595247 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:57.595094    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d5e532-2f4a-4165-82a0-630ac487ca87-tls-certs\") pod \"a1d5e532-2f4a-4165-82a0-630ac487ca87\" (UID: \"a1d5e532-2f4a-4165-82a0-630ac487ca87\") "
Sep 17 15:53:57.597211 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:57.597182    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1d5e532-2f4a-4165-82a0-630ac487ca87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1d5e532-2f4a-4165-82a0-630ac487ca87" (UID: "a1d5e532-2f4a-4165-82a0-630ac487ca87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:53:57.597336 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:57.597284    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d5e532-2f4a-4165-82a0-630ac487ca87-kube-api-access-ndc8x" (OuterVolumeSpecName: "kube-api-access-ndc8x") pod "a1d5e532-2f4a-4165-82a0-630ac487ca87" (UID: "a1d5e532-2f4a-4165-82a0-630ac487ca87"). InnerVolumeSpecName "kube-api-access-ndc8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:53:57.696708 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:57.696669    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d5e532-2f4a-4165-82a0-630ac487ca87-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:53:57.696708 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:57.696704    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndc8x\" (UniqueName: \"kubernetes.io/projected/a1d5e532-2f4a-4165-82a0-630ac487ca87-kube-api-access-ndc8x\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:53:58.214273 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:58.214243    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m_a1d5e532-2f4a-4165-82a0-630ac487ca87/main/0.log"
Sep 17 15:53:58.214471 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:58.214285    2574 generic.go:356] "Generic (PLEG): container finished" podID="a1d5e532-2f4a-4165-82a0-630ac487ca87" containerID="afd57fc52463abed1eb7a66b6cbc62765fe90df9dbadd128ef4254da9c46edc8" exitCode=1
Sep 17 15:53:58.214471 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:58.214335    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"
Sep 17 15:53:58.214471 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:58.214376    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m" event={"ID":"a1d5e532-2f4a-4165-82a0-630ac487ca87","Type":"ContainerDied","Data":"afd57fc52463abed1eb7a66b6cbc62765fe90df9dbadd128ef4254da9c46edc8"}
Sep 17 15:53:58.214471 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:58.214427    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m" event={"ID":"a1d5e532-2f4a-4165-82a0-630ac487ca87","Type":"ContainerDied","Data":"b3ebe9e18bedc213b0bb5d1f6faac6c2eca9c540f91e9d72884e9ebf20dc4667"}
Sep 17 15:53:58.214645 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:58.214474    2574 scope.go:117] "RemoveContainer" containerID="afd57fc52463abed1eb7a66b6cbc62765fe90df9dbadd128ef4254da9c46edc8"
Sep 17 15:53:58.223175 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:58.223155    2574 scope.go:117] "RemoveContainer" containerID="afd57fc52463abed1eb7a66b6cbc62765fe90df9dbadd128ef4254da9c46edc8"
Sep 17 15:53:58.223543 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:53:58.223516    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd57fc52463abed1eb7a66b6cbc62765fe90df9dbadd128ef4254da9c46edc8\": container with ID starting with afd57fc52463abed1eb7a66b6cbc62765fe90df9dbadd128ef4254da9c46edc8 not found: ID does not exist" containerID="afd57fc52463abed1eb7a66b6cbc62765fe90df9dbadd128ef4254da9c46edc8"
Sep 17 15:53:58.223599 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:58.223544    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd57fc52463abed1eb7a66b6cbc62765fe90df9dbadd128ef4254da9c46edc8"} err="failed to get container status \"afd57fc52463abed1eb7a66b6cbc62765fe90df9dbadd128ef4254da9c46edc8\": rpc error: code = NotFound desc = could not find container \"afd57fc52463abed1eb7a66b6cbc62765fe90df9dbadd128ef4254da9c46edc8\": container with ID starting with afd57fc52463abed1eb7a66b6cbc62765fe90df9dbadd128ef4254da9c46edc8 not found: ID does not exist"
Sep 17 15:53:58.240083 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:58.239313    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"]
Sep 17 15:53:58.244191 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:58.244155    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zfr4m"]
Sep 17 15:53:59.049718 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:53:59.049675    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1d5e532-2f4a-4165-82a0-630ac487ca87" path="/var/lib/kubelet/pods/a1d5e532-2f4a-4165-82a0-630ac487ca87/volumes"
Sep 17 15:54:00.883113 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:00.883068    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt" podUID="24d2316b-e4a3-48a5-905e-7ce7c66c5430" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:54:04.054992 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:04.054946    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt" podUID="24d2316b-e4a3-48a5-905e-7ce7c66c5430" containerName="main" containerID="cri-o://a666a2be8001bb80fe0be98f9c7eda268eb12f285cd63ffdd93d7f9c46543655" gracePeriod=45
Sep 17 15:54:10.883367 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:10.883316    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt" podUID="24d2316b-e4a3-48a5-905e-7ce7c66c5430" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:54:10.883925 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:10.883476    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"
Sep 17 15:54:20.883017 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:20.882972    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt" podUID="24d2316b-e4a3-48a5-905e-7ce7c66c5430" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:54:30.883561 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:30.883510    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt" podUID="24d2316b-e4a3-48a5-905e-7ce7c66c5430" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:54:34.310977 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.310953    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"
Sep 17 15:54:34.389941 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.389908    2574 generic.go:356] "Generic (PLEG): container finished" podID="24d2316b-e4a3-48a5-905e-7ce7c66c5430" containerID="a666a2be8001bb80fe0be98f9c7eda268eb12f285cd63ffdd93d7f9c46543655" exitCode=0
Sep 17 15:54:34.390120 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.389960    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"
Sep 17 15:54:34.390120 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.389985    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt" event={"ID":"24d2316b-e4a3-48a5-905e-7ce7c66c5430","Type":"ContainerDied","Data":"a666a2be8001bb80fe0be98f9c7eda268eb12f285cd63ffdd93d7f9c46543655"}
Sep 17 15:54:34.390120 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.390025    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt" event={"ID":"24d2316b-e4a3-48a5-905e-7ce7c66c5430","Type":"ContainerDied","Data":"87250448c3f317d3d0dbc0b6e9493e120cbbe693b502d297f1157194416e75a4"}
Sep 17 15:54:34.390120 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.390046    2574 scope.go:117] "RemoveContainer" containerID="a666a2be8001bb80fe0be98f9c7eda268eb12f285cd63ffdd93d7f9c46543655"
Sep 17 15:54:34.398364 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.398344    2574 scope.go:117] "RemoveContainer" containerID="a666a2be8001bb80fe0be98f9c7eda268eb12f285cd63ffdd93d7f9c46543655"
Sep 17 15:54:34.398664 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:54:34.398645    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a666a2be8001bb80fe0be98f9c7eda268eb12f285cd63ffdd93d7f9c46543655\": container with ID starting with a666a2be8001bb80fe0be98f9c7eda268eb12f285cd63ffdd93d7f9c46543655 not found: ID does not exist" containerID="a666a2be8001bb80fe0be98f9c7eda268eb12f285cd63ffdd93d7f9c46543655"
Sep 17 15:54:34.398717 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.398672    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a666a2be8001bb80fe0be98f9c7eda268eb12f285cd63ffdd93d7f9c46543655"} err="failed to get container status \"a666a2be8001bb80fe0be98f9c7eda268eb12f285cd63ffdd93d7f9c46543655\": rpc error: code = NotFound desc = could not find container \"a666a2be8001bb80fe0be98f9c7eda268eb12f285cd63ffdd93d7f9c46543655\": container with ID starting with a666a2be8001bb80fe0be98f9c7eda268eb12f285cd63ffdd93d7f9c46543655 not found: ID does not exist"
Sep 17 15:54:34.407948 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.407928    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2nhc\" (UniqueName: \"kubernetes.io/projected/24d2316b-e4a3-48a5-905e-7ce7c66c5430-kube-api-access-c2nhc\") pod \"24d2316b-e4a3-48a5-905e-7ce7c66c5430\" (UID: \"24d2316b-e4a3-48a5-905e-7ce7c66c5430\") "
Sep 17 15:54:34.408041 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.407969    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d2316b-e4a3-48a5-905e-7ce7c66c5430-tls-certs\") pod \"24d2316b-e4a3-48a5-905e-7ce7c66c5430\" (UID: \"24d2316b-e4a3-48a5-905e-7ce7c66c5430\") "
Sep 17 15:54:34.409982 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.409955    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d2316b-e4a3-48a5-905e-7ce7c66c5430-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24d2316b-e4a3-48a5-905e-7ce7c66c5430" (UID: "24d2316b-e4a3-48a5-905e-7ce7c66c5430"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:54:34.410074 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.410003    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d2316b-e4a3-48a5-905e-7ce7c66c5430-kube-api-access-c2nhc" (OuterVolumeSpecName: "kube-api-access-c2nhc") pod "24d2316b-e4a3-48a5-905e-7ce7c66c5430" (UID: "24d2316b-e4a3-48a5-905e-7ce7c66c5430"). InnerVolumeSpecName "kube-api-access-c2nhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:34.508827 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.508793    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2nhc\" (UniqueName: \"kubernetes.io/projected/24d2316b-e4a3-48a5-905e-7ce7c66c5430-kube-api-access-c2nhc\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:54:34.508827 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.508825    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d2316b-e4a3-48a5-905e-7ce7c66c5430-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:54:34.719479 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.719433    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"]
Sep 17 15:54:34.729161 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.729133    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-czvzt"]
Sep 17 15:54:34.805878 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.805843    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:34.914618 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:34.914582    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:35.048453 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:35.048368    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24d2316b-e4a3-48a5-905e-7ce7c66c5430" path="/var/lib/kubelet/pods/24d2316b-e4a3-48a5-905e-7ce7c66c5430/volumes"
Sep 17 15:54:38.960796 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:38.960758    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:39.087899 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:39.087858    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:42.722063 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:42.722026    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:46.539132 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.539089    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:46.591183 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.591150    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"]
Sep 17 15:54:46.591544 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.591531    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d5e532-2f4a-4165-82a0-630ac487ca87" containerName="main"
Sep 17 15:54:46.591597 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.591545    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d5e532-2f4a-4165-82a0-630ac487ca87" containerName="main"
Sep 17 15:54:46.591597 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.591555    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5" containerName="main"
Sep 17 15:54:46.591597 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.591562    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5" containerName="main"
Sep 17 15:54:46.591597 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.591568    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d2316b-e4a3-48a5-905e-7ce7c66c5430" containerName="main"
Sep 17 15:54:46.591597 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.591573    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d2316b-e4a3-48a5-905e-7ce7c66c5430" containerName="main"
Sep 17 15:54:46.591747 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.591640    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="24d2316b-e4a3-48a5-905e-7ce7c66c5430" containerName="main"
Sep 17 15:54:46.591747 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.591647    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1d5e532-2f4a-4165-82a0-630ac487ca87" containerName="main"
Sep 17 15:54:46.591747 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.591655    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ba0ec0-d6a8-4732-9ae0-8d82e9eaf7f5" containerName="main"
Sep 17 15:54:46.595998 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.595981    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.599115 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.599095    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-rhgkf\""
Sep 17 15:54:46.599243 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.599137    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 15:54:46.600235 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.600215    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 15:54:46.600235 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.600228    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 15:54:46.606802 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.606781    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"]
Sep 17 15:54:46.675077 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.675039    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:46.715167 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.715131    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dff479b-6d8b-448e-9d69-23887370b7c3-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.715335 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.715196    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.715335 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.715223    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6n6\" (UniqueName: \"kubernetes.io/projected/6dff479b-6d8b-448e-9d69-23887370b7c3-kube-api-access-gt6n6\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.715335 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.715267    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.715488 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.715331    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.715488 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.715366    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.816586 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.816489    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.816586 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.816533    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.816586 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.816566    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.816856 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.816611    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dff479b-6d8b-448e-9d69-23887370b7c3-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.816856 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.816659    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.816856 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.816683    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6n6\" (UniqueName: \"kubernetes.io/projected/6dff479b-6d8b-448e-9d69-23887370b7c3-kube-api-access-gt6n6\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.817034 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.817008    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.817099 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.817055    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.817099 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.817089    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.818816 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.818786    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.819148 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.819128    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dff479b-6d8b-448e-9d69-23887370b7c3-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.824946 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.824928    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6n6\" (UniqueName: \"kubernetes.io/projected/6dff479b-6d8b-448e-9d69-23887370b7c3-kube-api-access-gt6n6\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-4c4nv\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.907245 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.907208    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:46.913737 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.913704    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"]
Sep 17 15:54:46.917931 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.917910    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:46.920840 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.920820    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-6dklp\""
Sep 17 15:54:46.926005 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.925981    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"]
Sep 17 15:54:46.943858 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:46.943811    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:47.017779 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.017748    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa8bee69-caa1-4582-92ad-db1b0d527b59-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:47.017938 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.017788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa8bee69-caa1-4582-92ad-db1b0d527b59-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:47.017938 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.017851    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8bee69-caa1-4582-92ad-db1b0d527b59-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:47.017938 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.017909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t24qj\" (UniqueName: \"kubernetes.io/projected/aa8bee69-caa1-4582-92ad-db1b0d527b59-kube-api-access-t24qj\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:47.118374 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.118342    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa8bee69-caa1-4582-92ad-db1b0d527b59-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:47.118584 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.118395    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8bee69-caa1-4582-92ad-db1b0d527b59-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:47.118584 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.118463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t24qj\" (UniqueName: \"kubernetes.io/projected/aa8bee69-caa1-4582-92ad-db1b0d527b59-kube-api-access-t24qj\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:47.118584 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.118556    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa8bee69-caa1-4582-92ad-db1b0d527b59-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:47.118836 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.118814    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa8bee69-caa1-4582-92ad-db1b0d527b59-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:47.118899 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.118862    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa8bee69-caa1-4582-92ad-db1b0d527b59-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:47.121020 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.121001    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8bee69-caa1-4582-92ad-db1b0d527b59-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:47.128304 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.128281    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t24qj\" (UniqueName: \"kubernetes.io/projected/aa8bee69-caa1-4582-92ad-db1b0d527b59-kube-api-access-t24qj\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:47.188424 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.188393    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"]
Sep 17 15:54:47.189970 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:54:47.189931    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dff479b_6d8b_448e_9d69_23887370b7c3.slice/crio-2c5002160b5bb2d2e02ef0fa85fc9370eb0aa2d1f17699de422eda24709a0c3e WatchSource:0}: Error finding container 2c5002160b5bb2d2e02ef0fa85fc9370eb0aa2d1f17699de422eda24709a0c3e: Status 404 returned error can't find the container with id 2c5002160b5bb2d2e02ef0fa85fc9370eb0aa2d1f17699de422eda24709a0c3e
Sep 17 15:54:47.242603 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.242581    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:47.458666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.458564    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv" event={"ID":"6dff479b-6d8b-448e-9d69-23887370b7c3","Type":"ContainerStarted","Data":"9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed"}
Sep 17 15:54:47.458666 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.458638    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv" event={"ID":"6dff479b-6d8b-448e-9d69-23887370b7c3","Type":"ContainerStarted","Data":"2c5002160b5bb2d2e02ef0fa85fc9370eb0aa2d1f17699de422eda24709a0c3e"}
Sep 17 15:54:47.484879 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.484831    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv" podStartSLOduration=1.484814491 podStartE2EDuration="1.484814491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:47.48289479 +0000 UTC m=+952.954734640" watchObservedRunningTime="2026-09-17 15:54:47.484814491 +0000 UTC m=+952.956654337"
Sep 17 15:54:47.521707 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.521681    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"]
Sep 17 15:54:47.524395 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:54:47.524371    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8bee69_caa1_4582_92ad_db1b0d527b59.slice/crio-1a7b51c2405458fc76a68f9076e7110334caac2a5e328fe4e10d347972b4dddd WatchSource:0}: Error finding container 1a7b51c2405458fc76a68f9076e7110334caac2a5e328fe4e10d347972b4dddd: Status 404 returned error can't find the container with id 1a7b51c2405458fc76a68f9076e7110334caac2a5e328fe4e10d347972b4dddd
Sep 17 15:54:47.907710 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:47.907671    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:48.464633 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:48.464590    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6" event={"ID":"aa8bee69-caa1-4582-92ad-db1b0d527b59","Type":"ContainerStarted","Data":"3d62cc7a5693f9d65a6a9784badcf0c14cd8155f7a47679beb209ab2c2b28fe6"}
Sep 17 15:54:48.464823 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:48.464776    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:54:48.464823 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:48.464797    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6" event={"ID":"aa8bee69-caa1-4582-92ad-db1b0d527b59","Type":"ContainerStarted","Data":"1a7b51c2405458fc76a68f9076e7110334caac2a5e328fe4e10d347972b4dddd"}
Sep 17 15:54:48.486758 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:48.486709    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6" podStartSLOduration=2.486691526 podStartE2EDuration="2.486691526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:48.484790385 +0000 UTC m=+953.956630231" watchObservedRunningTime="2026-09-17 15:54:48.486691526 +0000 UTC m=+953.958531375"
Sep 17 15:54:49.389260 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:49.389227    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:49.490662 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:49.490627    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:52.582162 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:52.582119    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:52.682276 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:52.682242    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:56.908201 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:56.908158    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:56.920526 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:56.920494    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:54:57.523653 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:54:57.523619    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:55:19.472235 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:19.472157    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:55:19.482636 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:19.482603    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:24.049383 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:24.049346    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:29.693518 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:29.693477    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:29.947986 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:29.947899    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"]
Sep 17 15:55:29.954543 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:29.954511    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"
Sep 17 15:55:29.959058 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:29.959025    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 15:55:29.959211 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:29.959096    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-jppsv\""
Sep 17 15:55:29.959283 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:29.959235    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-fngcw\""
Sep 17 15:55:29.959345 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:29.959334    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 15:55:29.959420 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:29.959404    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 15:55:29.967102 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:29.967076    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"]
Sep 17 15:55:29.981867 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:29.981824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzw2f\" (UniqueName: \"kubernetes.io/projected/3e6f596c-162a-47cb-9110-9efbe2544d26-kube-api-access-tzw2f\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x\" (UID: \"3e6f596c-162a-47cb-9110-9efbe2544d26\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"
Sep 17 15:55:29.982032 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:29.981954    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6f596c-162a-47cb-9110-9efbe2544d26-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x\" (UID: \"3e6f596c-162a-47cb-9110-9efbe2544d26\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"
Sep 17 15:55:30.083056 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:30.083016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6f596c-162a-47cb-9110-9efbe2544d26-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x\" (UID: \"3e6f596c-162a-47cb-9110-9efbe2544d26\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"
Sep 17 15:55:30.083264 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:30.083095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzw2f\" (UniqueName: \"kubernetes.io/projected/3e6f596c-162a-47cb-9110-9efbe2544d26-kube-api-access-tzw2f\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x\" (UID: \"3e6f596c-162a-47cb-9110-9efbe2544d26\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"
Sep 17 15:55:30.085651 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:30.085626    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6f596c-162a-47cb-9110-9efbe2544d26-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x\" (UID: \"3e6f596c-162a-47cb-9110-9efbe2544d26\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"
Sep 17 15:55:30.092020 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:30.091997    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzw2f\" (UniqueName: \"kubernetes.io/projected/3e6f596c-162a-47cb-9110-9efbe2544d26-kube-api-access-tzw2f\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x\" (UID: \"3e6f596c-162a-47cb-9110-9efbe2544d26\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"
Sep 17 15:55:30.266577 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:30.266479    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"
Sep 17 15:55:30.581914 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:30.581882    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"]
Sep 17 15:55:30.583787 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:55:30.583762    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6f596c_162a_47cb_9110_9efbe2544d26.slice/crio-9af72b7a82dadb8b9005bac4e46437274551b41c524ffb611cce7ef70464ac17 WatchSource:0}: Error finding container 9af72b7a82dadb8b9005bac4e46437274551b41c524ffb611cce7ef70464ac17: Status 404 returned error can't find the container with id 9af72b7a82dadb8b9005bac4e46437274551b41c524ffb611cce7ef70464ac17
Sep 17 15:55:30.677116 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:30.677077    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x" event={"ID":"3e6f596c-162a-47cb-9110-9efbe2544d26","Type":"ContainerStarted","Data":"0d5d39ee9671ebfd2c93ec8260bfec2cbab46fded232aacb2faecc741a3f2140"}
Sep 17 15:55:30.677290 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:30.677145    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x" event={"ID":"3e6f596c-162a-47cb-9110-9efbe2544d26","Type":"ContainerStarted","Data":"9af72b7a82dadb8b9005bac4e46437274551b41c524ffb611cce7ef70464ac17"}
Sep 17 15:55:30.677290 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:30.677288    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"
Sep 17 15:55:30.703418 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:30.703369    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x" podStartSLOduration=1.7033493800000001 podStartE2EDuration="1.70334938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:30.700986138 +0000 UTC m=+996.172825985" watchObservedRunningTime="2026-09-17 15:55:30.70334938 +0000 UTC m=+996.175189226"
Sep 17 15:55:32.537706 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:32.537661    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:45.060422 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:45.060388    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:45.414551 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:45.414514    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:47.146755 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.146718    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:47.166191 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.166145    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"]
Sep 17 15:55:47.168475 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.168431    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"]
Sep 17 15:55:47.176860 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:55:47.176817    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 15:55:47.176860 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:55:47.176860 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 15:55:47.176860 ip-10-0-129-37 kubenswrapper[2574]:  > containerID="9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed" cmd=["/bin/sleep","15"]
Sep 17 15:55:47.177173 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.176878    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv" message="" err=<
Sep 17 15:55:47.177173 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:55:47.177173 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 15:55:47.177173 ip-10-0-129-37 kubenswrapper[2574]:  >
Sep 17 15:55:47.177173 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:55:47.176891    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 15:55:47.177173 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:55:47.177173 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 15:55:47.177173 ip-10-0-129-37 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv" podUID="6dff479b-6d8b-448e-9d69-23887370b7c3" containerName="main" containerID="cri-o://9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed"
Sep 17 15:55:47.177173 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.176927    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv" podUID="6dff479b-6d8b-448e-9d69-23887370b7c3" containerName="main" containerID="cri-o://9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed" gracePeriod=60
Sep 17 15:55:47.222056 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.222005    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:47.526487 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.526462    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:55:47.653287 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.653250    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-home\") pod \"6dff479b-6d8b-448e-9d69-23887370b7c3\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") "
Sep 17 15:55:47.653497 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.653300    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-dshm\") pod \"6dff479b-6d8b-448e-9d69-23887370b7c3\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") "
Sep 17 15:55:47.653497 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.653352    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dff479b-6d8b-448e-9d69-23887370b7c3-tls-certs\") pod \"6dff479b-6d8b-448e-9d69-23887370b7c3\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") "
Sep 17 15:55:47.653497 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.653396    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-model-cache\") pod \"6dff479b-6d8b-448e-9d69-23887370b7c3\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") "
Sep 17 15:55:47.653497 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.653434    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt6n6\" (UniqueName: \"kubernetes.io/projected/6dff479b-6d8b-448e-9d69-23887370b7c3-kube-api-access-gt6n6\") pod \"6dff479b-6d8b-448e-9d69-23887370b7c3\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") "
Sep 17 15:55:47.653497 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.653474    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-tmp-dir\") pod \"6dff479b-6d8b-448e-9d69-23887370b7c3\" (UID: \"6dff479b-6d8b-448e-9d69-23887370b7c3\") "
Sep 17 15:55:47.653799 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.653620    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-home" (OuterVolumeSpecName: "home") pod "6dff479b-6d8b-448e-9d69-23887370b7c3" (UID: "6dff479b-6d8b-448e-9d69-23887370b7c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:55:47.653799 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.653718    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "6dff479b-6d8b-448e-9d69-23887370b7c3" (UID: "6dff479b-6d8b-448e-9d69-23887370b7c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:55:47.653920 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.653829    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6dff479b-6d8b-448e-9d69-23887370b7c3" (UID: "6dff479b-6d8b-448e-9d69-23887370b7c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:55:47.653920 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.653882    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:55:47.653920 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.653898    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:55:47.655740 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.655714    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dff479b-6d8b-448e-9d69-23887370b7c3-kube-api-access-gt6n6" (OuterVolumeSpecName: "kube-api-access-gt6n6") pod "6dff479b-6d8b-448e-9d69-23887370b7c3" (UID: "6dff479b-6d8b-448e-9d69-23887370b7c3"). InnerVolumeSpecName "kube-api-access-gt6n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:55:47.655851 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.655749    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dff479b-6d8b-448e-9d69-23887370b7c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6dff479b-6d8b-448e-9d69-23887370b7c3" (UID: "6dff479b-6d8b-448e-9d69-23887370b7c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:55:47.655851 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.655757    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-dshm" (OuterVolumeSpecName: "dshm") pod "6dff479b-6d8b-448e-9d69-23887370b7c3" (UID: "6dff479b-6d8b-448e-9d69-23887370b7c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:55:47.754463 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.754370    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt6n6\" (UniqueName: \"kubernetes.io/projected/6dff479b-6d8b-448e-9d69-23887370b7c3-kube-api-access-gt6n6\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:55:47.754463 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.754396    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:55:47.754463 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.754406    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dff479b-6d8b-448e-9d69-23887370b7c3-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:55:47.754463 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.754415    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dff479b-6d8b-448e-9d69-23887370b7c3-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:55:47.760590 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.760554    2574 generic.go:356] "Generic (PLEG): container finished" podID="6dff479b-6d8b-448e-9d69-23887370b7c3" containerID="9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed" exitCode=0
Sep 17 15:55:47.760725 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.760595    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"
Sep 17 15:55:47.760725 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.760639    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv" event={"ID":"6dff479b-6d8b-448e-9d69-23887370b7c3","Type":"ContainerDied","Data":"9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed"}
Sep 17 15:55:47.760725 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.760694    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv" event={"ID":"6dff479b-6d8b-448e-9d69-23887370b7c3","Type":"ContainerDied","Data":"2c5002160b5bb2d2e02ef0fa85fc9370eb0aa2d1f17699de422eda24709a0c3e"}
Sep 17 15:55:47.760725 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.760719    2574 scope.go:117] "RemoveContainer" containerID="9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed"
Sep 17 15:55:47.770138 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.770122    2574 scope.go:117] "RemoveContainer" containerID="9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed"
Sep 17 15:55:47.770394 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:55:47.770380    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed\": container with ID starting with 9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed not found: ID does not exist" containerID="9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed"
Sep 17 15:55:47.770491 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.770401    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed"} err="failed to get container status \"9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed\": rpc error: code = NotFound desc = could not find container \"9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed\": container with ID starting with 9482600a72718a1baa09cf269456c7aec7f1179152feda284765b7d18b48d4ed not found: ID does not exist"
Sep 17 15:55:47.782657 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.782631    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"]
Sep 17 15:55:47.789223 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:47.789202    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv"]
Sep 17 15:55:48.513611 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:48.513566    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-4c4nv" podUID="6dff479b-6d8b-448e-9d69-23887370b7c3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 15:55:49.049149 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:49.049110    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dff479b-6d8b-448e-9d69-23887370b7c3" path="/var/lib/kubelet/pods/6dff479b-6d8b-448e-9d69-23887370b7c3/volumes"
Sep 17 15:55:49.473005 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:49.472965    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6" podUID="aa8bee69-caa1-4582-92ad-db1b0d527b59" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:55:59.472333 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:55:59.472291    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6" podUID="aa8bee69-caa1-4582-92ad-db1b0d527b59" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:56:01.684212 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:01.684178    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"
Sep 17 15:56:02.189587 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:02.189551    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6" podUID="aa8bee69-caa1-4582-92ad-db1b0d527b59" containerName="main" containerID="cri-o://3d62cc7a5693f9d65a6a9784badcf0c14cd8155f7a47679beb209ab2c2b28fe6" gracePeriod=45
Sep 17 15:56:08.678198 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:08.678161    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:08.691539 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:08.691515    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"]
Sep 17 15:56:08.795757 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:08.795725    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:09.473268 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:09.473232    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6" podUID="aa8bee69-caa1-4582-92ad-db1b0d527b59" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:56:09.473518 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:09.473339    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:56:11.683841 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:11.683802    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x" podUID="3e6f596c-162a-47cb-9110-9efbe2544d26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:56:19.472312 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:19.472270    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6" podUID="aa8bee69-caa1-4582-92ad-db1b0d527b59" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:56:21.683798 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:21.683757    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x" podUID="3e6f596c-162a-47cb-9110-9efbe2544d26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:56:23.710487 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:23.710430    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x" podUID="3e6f596c-162a-47cb-9110-9efbe2544d26" containerName="main" containerID="cri-o://0d5d39ee9671ebfd2c93ec8260bfec2cbab46fded232aacb2faecc741a3f2140" gracePeriod=45
Sep 17 15:56:29.472633 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:29.472585    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6" podUID="aa8bee69-caa1-4582-92ad-db1b0d527b59" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:56:30.618654 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:30.618619    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:31.683995 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:31.683956    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x" podUID="3e6f596c-162a-47cb-9110-9efbe2544d26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:56:31.684390 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:31.684073    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"
Sep 17 15:56:32.437556 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.437528    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:56:32.555429 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.555331    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa8bee69-caa1-4582-92ad-db1b0d527b59-tokenizer-cache\") pod \"aa8bee69-caa1-4582-92ad-db1b0d527b59\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") "
Sep 17 15:56:32.555429 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.555377    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t24qj\" (UniqueName: \"kubernetes.io/projected/aa8bee69-caa1-4582-92ad-db1b0d527b59-kube-api-access-t24qj\") pod \"aa8bee69-caa1-4582-92ad-db1b0d527b59\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") "
Sep 17 15:56:32.555429 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.555405    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa8bee69-caa1-4582-92ad-db1b0d527b59-cachi2-cache\") pod \"aa8bee69-caa1-4582-92ad-db1b0d527b59\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") "
Sep 17 15:56:32.555739 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.555502    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8bee69-caa1-4582-92ad-db1b0d527b59-tls-certs\") pod \"aa8bee69-caa1-4582-92ad-db1b0d527b59\" (UID: \"aa8bee69-caa1-4582-92ad-db1b0d527b59\") "
Sep 17 15:56:32.555739 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.555661    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8bee69-caa1-4582-92ad-db1b0d527b59-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "aa8bee69-caa1-4582-92ad-db1b0d527b59" (UID: "aa8bee69-caa1-4582-92ad-db1b0d527b59"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:32.555739 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.555667    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa8bee69-caa1-4582-92ad-db1b0d527b59-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "aa8bee69-caa1-4582-92ad-db1b0d527b59" (UID: "aa8bee69-caa1-4582-92ad-db1b0d527b59"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:32.555886 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.555863    2574 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa8bee69-caa1-4582-92ad-db1b0d527b59-tokenizer-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:56:32.555924 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.555891    2574 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa8bee69-caa1-4582-92ad-db1b0d527b59-cachi2-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:56:32.557605 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.557582    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa8bee69-caa1-4582-92ad-db1b0d527b59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa8bee69-caa1-4582-92ad-db1b0d527b59" (UID: "aa8bee69-caa1-4582-92ad-db1b0d527b59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:32.557673 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.557596    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8bee69-caa1-4582-92ad-db1b0d527b59-kube-api-access-t24qj" (OuterVolumeSpecName: "kube-api-access-t24qj") pod "aa8bee69-caa1-4582-92ad-db1b0d527b59" (UID: "aa8bee69-caa1-4582-92ad-db1b0d527b59"). InnerVolumeSpecName "kube-api-access-t24qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:32.656602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.656558    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t24qj\" (UniqueName: \"kubernetes.io/projected/aa8bee69-caa1-4582-92ad-db1b0d527b59-kube-api-access-t24qj\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:56:32.656602 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.656595    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8bee69-caa1-4582-92ad-db1b0d527b59-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:56:32.978778 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.978743    2574 generic.go:356] "Generic (PLEG): container finished" podID="aa8bee69-caa1-4582-92ad-db1b0d527b59" containerID="3d62cc7a5693f9d65a6a9784badcf0c14cd8155f7a47679beb209ab2c2b28fe6" exitCode=0
Sep 17 15:56:32.979330 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.978804    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"
Sep 17 15:56:32.979330 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.978825    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6" event={"ID":"aa8bee69-caa1-4582-92ad-db1b0d527b59","Type":"ContainerDied","Data":"3d62cc7a5693f9d65a6a9784badcf0c14cd8155f7a47679beb209ab2c2b28fe6"}
Sep 17 15:56:32.979330 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.978919    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6" event={"ID":"aa8bee69-caa1-4582-92ad-db1b0d527b59","Type":"ContainerDied","Data":"1a7b51c2405458fc76a68f9076e7110334caac2a5e328fe4e10d347972b4dddd"}
Sep 17 15:56:32.979330 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.978944    2574 scope.go:117] "RemoveContainer" containerID="3d62cc7a5693f9d65a6a9784badcf0c14cd8155f7a47679beb209ab2c2b28fe6"
Sep 17 15:56:32.988873 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.988855    2574 scope.go:117] "RemoveContainer" containerID="3d62cc7a5693f9d65a6a9784badcf0c14cd8155f7a47679beb209ab2c2b28fe6"
Sep 17 15:56:32.989145 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:56:32.989128    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d62cc7a5693f9d65a6a9784badcf0c14cd8155f7a47679beb209ab2c2b28fe6\": container with ID starting with 3d62cc7a5693f9d65a6a9784badcf0c14cd8155f7a47679beb209ab2c2b28fe6 not found: ID does not exist" containerID="3d62cc7a5693f9d65a6a9784badcf0c14cd8155f7a47679beb209ab2c2b28fe6"
Sep 17 15:56:32.989193 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:32.989153    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d62cc7a5693f9d65a6a9784badcf0c14cd8155f7a47679beb209ab2c2b28fe6"} err="failed to get container status \"3d62cc7a5693f9d65a6a9784badcf0c14cd8155f7a47679beb209ab2c2b28fe6\": rpc error: code = NotFound desc = could not find container \"3d62cc7a5693f9d65a6a9784badcf0c14cd8155f7a47679beb209ab2c2b28fe6\": container with ID starting with 3d62cc7a5693f9d65a6a9784badcf0c14cd8155f7a47679beb209ab2c2b28fe6 not found: ID does not exist"
Sep 17 15:56:33.001031 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:33.000999    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"]
Sep 17 15:56:33.008157 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:33.008130    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcstkm6"]
Sep 17 15:56:33.049818 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:33.049783    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa8bee69-caa1-4582-92ad-db1b0d527b59" path="/var/lib/kubelet/pods/aa8bee69-caa1-4582-92ad-db1b0d527b59/volumes"
Sep 17 15:56:34.320945 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:34.320903    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:34.418185 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:34.418152    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:41.683877 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:41.683839    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x" podUID="3e6f596c-162a-47cb-9110-9efbe2544d26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:56:41.790009 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:41.789977    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:48.895620 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:48.895538    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:48.990830 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:48.990797    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:51.683616 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:51.683577    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x" podUID="3e6f596c-162a-47cb-9110-9efbe2544d26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:56:53.967574 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:53.967514    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"
Sep 17 15:56:54.047702 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.047669    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6f596c-162a-47cb-9110-9efbe2544d26-tls-certs\") pod \"3e6f596c-162a-47cb-9110-9efbe2544d26\" (UID: \"3e6f596c-162a-47cb-9110-9efbe2544d26\") "
Sep 17 15:56:54.047883 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.047728    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzw2f\" (UniqueName: \"kubernetes.io/projected/3e6f596c-162a-47cb-9110-9efbe2544d26-kube-api-access-tzw2f\") pod \"3e6f596c-162a-47cb-9110-9efbe2544d26\" (UID: \"3e6f596c-162a-47cb-9110-9efbe2544d26\") "
Sep 17 15:56:54.049908 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.049876    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6f596c-162a-47cb-9110-9efbe2544d26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e6f596c-162a-47cb-9110-9efbe2544d26" (UID: "3e6f596c-162a-47cb-9110-9efbe2544d26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:54.050017 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.049923    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6f596c-162a-47cb-9110-9efbe2544d26-kube-api-access-tzw2f" (OuterVolumeSpecName: "kube-api-access-tzw2f") pod "3e6f596c-162a-47cb-9110-9efbe2544d26" (UID: "3e6f596c-162a-47cb-9110-9efbe2544d26"). InnerVolumeSpecName "kube-api-access-tzw2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:54.078110 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.078076    2574 generic.go:356] "Generic (PLEG): container finished" podID="3e6f596c-162a-47cb-9110-9efbe2544d26" containerID="0d5d39ee9671ebfd2c93ec8260bfec2cbab46fded232aacb2faecc741a3f2140" exitCode=0
Sep 17 15:56:54.078239 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.078122    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"
Sep 17 15:56:54.078239 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.078155    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x" event={"ID":"3e6f596c-162a-47cb-9110-9efbe2544d26","Type":"ContainerDied","Data":"0d5d39ee9671ebfd2c93ec8260bfec2cbab46fded232aacb2faecc741a3f2140"}
Sep 17 15:56:54.078239 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.078206    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x" event={"ID":"3e6f596c-162a-47cb-9110-9efbe2544d26","Type":"ContainerDied","Data":"9af72b7a82dadb8b9005bac4e46437274551b41c524ffb611cce7ef70464ac17"}
Sep 17 15:56:54.078239 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.078230    2574 scope.go:117] "RemoveContainer" containerID="0d5d39ee9671ebfd2c93ec8260bfec2cbab46fded232aacb2faecc741a3f2140"
Sep 17 15:56:54.086956 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.086939    2574 scope.go:117] "RemoveContainer" containerID="0d5d39ee9671ebfd2c93ec8260bfec2cbab46fded232aacb2faecc741a3f2140"
Sep 17 15:56:54.087253 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:56:54.087235    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5d39ee9671ebfd2c93ec8260bfec2cbab46fded232aacb2faecc741a3f2140\": container with ID starting with 0d5d39ee9671ebfd2c93ec8260bfec2cbab46fded232aacb2faecc741a3f2140 not found: ID does not exist" containerID="0d5d39ee9671ebfd2c93ec8260bfec2cbab46fded232aacb2faecc741a3f2140"
Sep 17 15:56:54.087300 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.087261    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5d39ee9671ebfd2c93ec8260bfec2cbab46fded232aacb2faecc741a3f2140"} err="failed to get container status \"0d5d39ee9671ebfd2c93ec8260bfec2cbab46fded232aacb2faecc741a3f2140\": rpc error: code = NotFound desc = could not find container \"0d5d39ee9671ebfd2c93ec8260bfec2cbab46fded232aacb2faecc741a3f2140\": container with ID starting with 0d5d39ee9671ebfd2c93ec8260bfec2cbab46fded232aacb2faecc741a3f2140 not found: ID does not exist"
Sep 17 15:56:54.100565 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.100544    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"]
Sep 17 15:56:54.106859 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.106838    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-lqp9x"]
Sep 17 15:56:54.149018 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.148994    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6f596c-162a-47cb-9110-9efbe2544d26-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:56:54.149018 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.149015    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzw2f\" (UniqueName: \"kubernetes.io/projected/3e6f596c-162a-47cb-9110-9efbe2544d26-kube-api-access-tzw2f\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:56:54.187056 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.187020    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"]
Sep 17 15:56:54.187469 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.187431    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e6f596c-162a-47cb-9110-9efbe2544d26" containerName="main"
Sep 17 15:56:54.187469 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.187464    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e6f596c-162a-47cb-9110-9efbe2544d26" containerName="main"
Sep 17 15:56:54.187623 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.187480    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dff479b-6d8b-448e-9d69-23887370b7c3" containerName="main"
Sep 17 15:56:54.187623 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.187486    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dff479b-6d8b-448e-9d69-23887370b7c3" containerName="main"
Sep 17 15:56:54.187623 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.187495    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa8bee69-caa1-4582-92ad-db1b0d527b59" containerName="main"
Sep 17 15:56:54.187623 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.187501    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa8bee69-caa1-4582-92ad-db1b0d527b59" containerName="main"
Sep 17 15:56:54.187623 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.187578    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e6f596c-162a-47cb-9110-9efbe2544d26" containerName="main"
Sep 17 15:56:54.187623 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.187588    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dff479b-6d8b-448e-9d69-23887370b7c3" containerName="main"
Sep 17 15:56:54.187623 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.187595    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa8bee69-caa1-4582-92ad-db1b0d527b59" containerName="main"
Sep 17 15:56:54.192606 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.192578    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.195005 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.194979    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 15:56:54.195155 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.195059    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 15:56:54.195432 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.195408    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-zvx8v\""
Sep 17 15:56:54.195588 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.195435    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-68wcj\""
Sep 17 15:56:54.195588 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.195500    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 15:56:54.212222 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.212192    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"]
Sep 17 15:56:54.244435 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.244346    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:54.249492 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.249415    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.249492 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.249476    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778a1232-40ce-4a2f-b2e6-ee8ab5389889-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.249664 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.249515    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.249664 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.249546    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vsxb\" (UniqueName: \"kubernetes.io/projected/778a1232-40ce-4a2f-b2e6-ee8ab5389889-kube-api-access-8vsxb\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.249664 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.249602    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.249664 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.249638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.350855 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.350819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.350855 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.350870    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.351106 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.350889    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778a1232-40ce-4a2f-b2e6-ee8ab5389889-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.351106 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.350931    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.351106 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.350971    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vsxb\" (UniqueName: \"kubernetes.io/projected/778a1232-40ce-4a2f-b2e6-ee8ab5389889-kube-api-access-8vsxb\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.351106 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.351006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.351351 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.351320    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.351802 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.351332    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.351802 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.351471    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.353297 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.353279    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.353634 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.353614    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778a1232-40ce-4a2f-b2e6-ee8ab5389889-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.359204 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.359179    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vsxb\" (UniqueName: \"kubernetes.io/projected/778a1232-40ce-4a2f-b2e6-ee8ab5389889-kube-api-access-8vsxb\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.433007 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.432964    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"]
Sep 17 15:56:54.436650 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.436625    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"
Sep 17 15:56:54.438985 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.438958    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-q8gsq\""
Sep 17 15:56:54.446591 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.446561    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"]
Sep 17 15:56:54.452470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.452420    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7bh2\" (UniqueName: \"kubernetes.io/projected/e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44-kube-api-access-k7bh2\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn\" (UID: \"e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"
Sep 17 15:56:54.452595 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.452548    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn\" (UID: \"e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"
Sep 17 15:56:54.482252 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.482220    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:54.505831 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.505727    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:54.553115 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.553072    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bh2\" (UniqueName: \"kubernetes.io/projected/e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44-kube-api-access-k7bh2\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn\" (UID: \"e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"
Sep 17 15:56:54.553299 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.553163    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn\" (UID: \"e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"
Sep 17 15:56:54.556701 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.556510    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn\" (UID: \"e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"
Sep 17 15:56:54.563272 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.563246    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bh2\" (UniqueName: \"kubernetes.io/projected/e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44-kube-api-access-k7bh2\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn\" (UID: \"e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"
Sep 17 15:56:54.747377 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.747337    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"
Sep 17 15:56:54.786372 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.786336    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"]
Sep 17 15:56:54.787884 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:56:54.787850    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778a1232_40ce_4a2f_b2e6_ee8ab5389889.slice/crio-4dd3864ffbde6e5ddb242e6f31ed79cc277c3dccd40781582cf5635ee2f9ac16 WatchSource:0}: Error finding container 4dd3864ffbde6e5ddb242e6f31ed79cc277c3dccd40781582cf5635ee2f9ac16: Status 404 returned error can't find the container with id 4dd3864ffbde6e5ddb242e6f31ed79cc277c3dccd40781582cf5635ee2f9ac16
Sep 17 15:56:54.790894 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:54.790853    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:56:55.025803 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:55.025716    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"]
Sep 17 15:56:55.028955 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:56:55.028925    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ecf6eb_c61b_4e8f_a77b_0606f1b48b44.slice/crio-6f2453c2a084f6b69bbdcfe3c7b39ed1d91f7592cea1a289a23cfeea591f103c WatchSource:0}: Error finding container 6f2453c2a084f6b69bbdcfe3c7b39ed1d91f7592cea1a289a23cfeea591f103c: Status 404 returned error can't find the container with id 6f2453c2a084f6b69bbdcfe3c7b39ed1d91f7592cea1a289a23cfeea591f103c
Sep 17 15:56:55.052546 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:55.052511    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e6f596c-162a-47cb-9110-9efbe2544d26" path="/var/lib/kubelet/pods/3e6f596c-162a-47cb-9110-9efbe2544d26/volumes"
Sep 17 15:56:55.085295 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:55.085262    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn" event={"ID":"e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44","Type":"ContainerStarted","Data":"6f2453c2a084f6b69bbdcfe3c7b39ed1d91f7592cea1a289a23cfeea591f103c"}
Sep 17 15:56:55.086589 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:55.086563    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt" event={"ID":"778a1232-40ce-4a2f-b2e6-ee8ab5389889","Type":"ContainerStarted","Data":"4dd3864ffbde6e5ddb242e6f31ed79cc277c3dccd40781582cf5635ee2f9ac16"}
Sep 17 15:56:56.092534 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:56.092486    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn" event={"ID":"e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44","Type":"ContainerStarted","Data":"5537106a3389cc49ac38096c3db3390576e94ebd615ed789dc244268eaa96708"}
Sep 17 15:56:56.092935 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:56.092591    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"
Sep 17 15:56:56.115690 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:56.115632    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn" podStartSLOduration=2.115612813 podStartE2EDuration="2.115612813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:56.112271634 +0000 UTC m=+1081.584111486" watchObservedRunningTime="2026-09-17 15:56:56.115612813 +0000 UTC m=+1081.587452659"
Sep 17 15:56:57.099150 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:57.099107    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt" event={"ID":"778a1232-40ce-4a2f-b2e6-ee8ab5389889","Type":"ContainerStarted","Data":"28e948660f3344dcc04f1dc1054d3e6170f59c0e3c32386a6c074e98afc8cdab"}
Sep 17 15:56:57.099675 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:57.099373    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:58.106075 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:58.106039    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt" event={"ID":"778a1232-40ce-4a2f-b2e6-ee8ab5389889","Type":"ContainerStarted","Data":"ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789"}
Sep 17 15:56:58.132814 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:58.132756    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt" podStartSLOduration=2.382487867 podStartE2EDuration="4.132734204s" totalImagesPullingTime="1.750246337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:58.132634792 +0000 UTC m=+1083.604474638" watchObservedRunningTime="2026-09-17 15:56:58.132734204 +0000 UTC m=+1083.604574122"
Sep 17 15:56:58.506905 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:58.506812    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:56:58.731633 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:58.731599    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:58.825077 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:58.824979    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:59.132430 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:59.132393    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:59.232418 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:56:59.232383    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:02.600709 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:02.600670    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:04.506644 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:04.506600    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:57:04.520596 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:04.520566    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:57:05.153048 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:05.153016    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:57:05.682781 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:05.682743    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:06.160698 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:06.160670    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:57:12.306106 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:12.306068    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:12.364726 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:12.364643    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:13.060136 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:13.060103    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:18.294775 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:18.294737    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:20.995406 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:20.995372    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:21.101321 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:21.101285    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:21.198296 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:21.198260    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:21.403382 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:21.403346    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:21.502973 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:21.502934    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:24.146953 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:24.146916    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:24.243239 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:24.243194    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:26.090614 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:26.090577    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:27.101051 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:27.101019    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"
Sep 17 15:57:31.541307 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:31.541269    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:33.011514 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:33.011476    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:38.386156 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:38.386114    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:38.926060 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:38.926026    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:39.031256 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:39.031211    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:46.516389 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:46.516355    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:48.788492 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:48.788422    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:48.880691 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:48.880659    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:50.852297 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:50.852257    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:53.856088 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:53.856049    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:59.506659 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:57:59.506614    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:00.043125 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:00.043084    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:07.656087 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.656056    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"]
Sep 17 15:58:07.660263 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.660239    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.662745 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.662717    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 17 15:58:07.662869 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.662749    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 15:58:07.662869 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.662796    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 15:58:07.662869 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.662825    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 15:58:07.662869 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.662829    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 15:58:07.663058 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.663038    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-2xcpn\""
Sep 17 15:58:07.673923 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.673897    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"]
Sep 17 15:58:07.692278 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.692240    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:07.696260 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.696227    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n5t4\" (UniqueName: \"kubernetes.io/projected/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-kube-api-access-7n5t4\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.696380 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.696293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-tmp-dir\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.696380 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.696315    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-cabundle-cert\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.696380 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.696351    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-model-cache\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.696380 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.696367    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-kserve-provision-location\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.696607 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.696411    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-home\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.696607 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.696491    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-dshm\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.696607 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.696536    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-tls-certs\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.797997 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.797960    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-dshm\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.797997 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.797996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-tls-certs\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.798256 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.798027    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n5t4\" (UniqueName: \"kubernetes.io/projected/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-kube-api-access-7n5t4\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.798256 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.798074    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-tmp-dir\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.798256 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.798089    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-cabundle-cert\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.798256 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.798108    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-model-cache\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.798256 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.798132    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-kserve-provision-location\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.798256 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.798188    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-home\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.798615 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.798570    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-home\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.798659 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.798634    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-tmp-dir\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.798729 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.798701    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-kserve-provision-location\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.798838 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.798821    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-model-cache\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.798884 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.798876    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-cabundle-cert\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.800852 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.800827    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-dshm\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.801248 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.801226    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-tls-certs\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.807112 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.807093    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n5t4\" (UniqueName: \"kubernetes.io/projected/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-kube-api-access-7n5t4\") pod \"auth-enabled-test-kserve-656ddb986c-l7h4k\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:07.970804 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:07.970691    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:08.063284 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:08.063248    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:08.252655 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:08.252628    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"]
Sep 17 15:58:08.254338 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:58:08.254306    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24aca2e_e52c_4cc4_b5b5_245d58585b2b.slice/crio-78b6e21adae5ef7c44850703a7eedb30b624eb72b8ba2ab71b9f1f976cc3fe22 WatchSource:0}: Error finding container 78b6e21adae5ef7c44850703a7eedb30b624eb72b8ba2ab71b9f1f976cc3fe22: Status 404 returned error can't find the container with id 78b6e21adae5ef7c44850703a7eedb30b624eb72b8ba2ab71b9f1f976cc3fe22
Sep 17 15:58:08.458866 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:08.458828    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" event={"ID":"e24aca2e-e52c-4cc4-b5b5-245d58585b2b","Type":"ContainerStarted","Data":"6ec24b30ea9a4b1c0f85095325f39fae2c15d97f390f90f33258a0dcb0e2bd38"}
Sep 17 15:58:08.459038 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:08.458881    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" event={"ID":"e24aca2e-e52c-4cc4-b5b5-245d58585b2b","Type":"ContainerStarted","Data":"78b6e21adae5ef7c44850703a7eedb30b624eb72b8ba2ab71b9f1f976cc3fe22"}
Sep 17 15:58:11.098583 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:11.098539    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:11.476414 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:11.476379    2574 generic.go:356] "Generic (PLEG): container finished" podID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerID="6ec24b30ea9a4b1c0f85095325f39fae2c15d97f390f90f33258a0dcb0e2bd38" exitCode=0
Sep 17 15:58:11.476626 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:11.476478    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" event={"ID":"e24aca2e-e52c-4cc4-b5b5-245d58585b2b","Type":"ContainerDied","Data":"6ec24b30ea9a4b1c0f85095325f39fae2c15d97f390f90f33258a0dcb0e2bd38"}
Sep 17 15:58:14.884504 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:14.884464    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:14.984458 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:14.984396    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:16.715470 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:16.715389    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:39.651962 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:39.651927    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" event={"ID":"e24aca2e-e52c-4cc4-b5b5-245d58585b2b","Type":"ContainerStarted","Data":"f479c401ab78bb76db9629d86199b275b5a3ee25738b340ce680ad31e0d4fe57"}
Sep 17 15:58:39.679541 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:39.679464    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" podStartSLOduration=2.515928942 podStartE2EDuration="32.679423762s" totalImagesPullingTime="27.812420461s" totalInitContainerRuntime="2.351074359s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:58:39.679150646 +0000 UTC m=+1185.150990492" watchObservedRunningTime="2026-09-17 15:58:39.679423762 +0000 UTC m=+1185.151263608"
Sep 17 15:58:39.971557 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:39.971475    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:47.971565 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:47.971527    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 15:58:47.973046 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:47.973016    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 15:58:57.345164 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:57.345124    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:57.353221 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:57.353189    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"]
Sep 17 15:58:57.361094 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:57.361063    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"]
Sep 17 15:58:57.364260 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:58:57.364225    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 15:58:57.364260 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:58:57.364260 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 15:58:57.364260 ip-10-0-129-37 kubenswrapper[2574]:  > containerID="ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789" cmd=["/bin/sleep","15"]
Sep 17 15:58:57.364587 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:57.364287    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt" message="" err=<
Sep 17 15:58:57.364587 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:58:57.364587 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 15:58:57.364587 ip-10-0-129-37 kubenswrapper[2574]:  >
Sep 17 15:58:57.364587 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:58:57.364304    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 15:58:57.364587 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 15:58:57.364587 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 15:58:57.364587 ip-10-0-129-37 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt" podUID="778a1232-40ce-4a2f-b2e6-ee8ab5389889" containerName="main" containerID="cri-o://ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789"
Sep 17 15:58:57.364587 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:57.364360    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt" podUID="778a1232-40ce-4a2f-b2e6-ee8ab5389889" containerName="main" containerID="cri-o://ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789" gracePeriod=60
Sep 17 15:58:57.446874 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:57.446838    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:57.753970 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:57.753922    2574 generic.go:356] "Generic (PLEG): container finished" podID="778a1232-40ce-4a2f-b2e6-ee8ab5389889" containerID="ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789" exitCode=0
Sep 17 15:58:57.754148 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:57.753965    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt" event={"ID":"778a1232-40ce-4a2f-b2e6-ee8ab5389889","Type":"ContainerDied","Data":"ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789"}
Sep 17 15:58:57.971762 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:57.971717    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 15:58:57.992948 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:57.992901    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt" podUID="778a1232-40ce-4a2f-b2e6-ee8ab5389889" containerName="llm-d-routing-sidecar" containerID="cri-o://28e948660f3344dcc04f1dc1054d3e6170f59c0e3c32386a6c074e98afc8cdab" gracePeriod=60
Sep 17 15:58:58.141811 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.141765    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt" podUID="778a1232-40ce-4a2f-b2e6-ee8ab5389889" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8001/health\": dial tcp 10.134.0.41:8001: connect: connection refused"
Sep 17 15:58:58.278748 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.278675    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:58:58.366337 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.366298    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-dshm\") pod \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") "
Sep 17 15:58:58.366844 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.366362    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-tmp-dir\") pod \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") "
Sep 17 15:58:58.366844 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.366386    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778a1232-40ce-4a2f-b2e6-ee8ab5389889-tls-certs\") pod \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") "
Sep 17 15:58:58.366844 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.366411    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vsxb\" (UniqueName: \"kubernetes.io/projected/778a1232-40ce-4a2f-b2e6-ee8ab5389889-kube-api-access-8vsxb\") pod \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") "
Sep 17 15:58:58.366844 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.366437    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-home\") pod \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") "
Sep 17 15:58:58.366844 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.366513    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-model-cache\") pod \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\" (UID: \"778a1232-40ce-4a2f-b2e6-ee8ab5389889\") "
Sep 17 15:58:58.367104 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.366961    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-model-cache" (OuterVolumeSpecName: "model-cache") pod "778a1232-40ce-4a2f-b2e6-ee8ab5389889" (UID: "778a1232-40ce-4a2f-b2e6-ee8ab5389889"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:58.367104 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.366976    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-home" (OuterVolumeSpecName: "home") pod "778a1232-40ce-4a2f-b2e6-ee8ab5389889" (UID: "778a1232-40ce-4a2f-b2e6-ee8ab5389889"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:58.367213 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.367176    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "778a1232-40ce-4a2f-b2e6-ee8ab5389889" (UID: "778a1232-40ce-4a2f-b2e6-ee8ab5389889"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:58.369389 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.369352    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-dshm" (OuterVolumeSpecName: "dshm") pod "778a1232-40ce-4a2f-b2e6-ee8ab5389889" (UID: "778a1232-40ce-4a2f-b2e6-ee8ab5389889"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:58.369575 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.369506    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778a1232-40ce-4a2f-b2e6-ee8ab5389889-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "778a1232-40ce-4a2f-b2e6-ee8ab5389889" (UID: "778a1232-40ce-4a2f-b2e6-ee8ab5389889"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:58:58.369651 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.369594    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778a1232-40ce-4a2f-b2e6-ee8ab5389889-kube-api-access-8vsxb" (OuterVolumeSpecName: "kube-api-access-8vsxb") pod "778a1232-40ce-4a2f-b2e6-ee8ab5389889" (UID: "778a1232-40ce-4a2f-b2e6-ee8ab5389889"). InnerVolumeSpecName "kube-api-access-8vsxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:58:58.467641 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.467597    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:58:58.467641 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.467633    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:58:58.467641 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.467645    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:58:58.467890 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.467655    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778a1232-40ce-4a2f-b2e6-ee8ab5389889-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:58:58.467890 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.467666    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vsxb\" (UniqueName: \"kubernetes.io/projected/778a1232-40ce-4a2f-b2e6-ee8ab5389889-kube-api-access-8vsxb\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:58:58.467890 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.467681    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778a1232-40ce-4a2f-b2e6-ee8ab5389889-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:58:58.761423 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.761381    2574 generic.go:356] "Generic (PLEG): container finished" podID="778a1232-40ce-4a2f-b2e6-ee8ab5389889" containerID="28e948660f3344dcc04f1dc1054d3e6170f59c0e3c32386a6c074e98afc8cdab" exitCode=0
Sep 17 15:58:58.761611 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.761465    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"
Sep 17 15:58:58.761611 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.761469    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt" event={"ID":"778a1232-40ce-4a2f-b2e6-ee8ab5389889","Type":"ContainerDied","Data":"28e948660f3344dcc04f1dc1054d3e6170f59c0e3c32386a6c074e98afc8cdab"}
Sep 17 15:58:58.761611 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.761530    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt" event={"ID":"778a1232-40ce-4a2f-b2e6-ee8ab5389889","Type":"ContainerDied","Data":"4dd3864ffbde6e5ddb242e6f31ed79cc277c3dccd40781582cf5635ee2f9ac16"}
Sep 17 15:58:58.761611 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.761556    2574 scope.go:117] "RemoveContainer" containerID="ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789"
Sep 17 15:58:58.772078 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.772056    2574 scope.go:117] "RemoveContainer" containerID="28e948660f3344dcc04f1dc1054d3e6170f59c0e3c32386a6c074e98afc8cdab"
Sep 17 15:58:58.782274 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.782252    2574 scope.go:117] "RemoveContainer" containerID="ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789"
Sep 17 15:58:58.783482 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:58:58.783433    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789\": container with ID starting with ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789 not found: ID does not exist" containerID="ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789"
Sep 17 15:58:58.783616 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.783494    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789"} err="failed to get container status \"ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789\": rpc error: code = NotFound desc = could not find container \"ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789\": container with ID starting with ee6a78736532093f7d3a7f008d7f44b6f0c2244b24dfc846dfb260d99c711789 not found: ID does not exist"
Sep 17 15:58:58.783616 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.783519    2574 scope.go:117] "RemoveContainer" containerID="28e948660f3344dcc04f1dc1054d3e6170f59c0e3c32386a6c074e98afc8cdab"
Sep 17 15:58:58.783859 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:58:58.783829    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e948660f3344dcc04f1dc1054d3e6170f59c0e3c32386a6c074e98afc8cdab\": container with ID starting with 28e948660f3344dcc04f1dc1054d3e6170f59c0e3c32386a6c074e98afc8cdab not found: ID does not exist" containerID="28e948660f3344dcc04f1dc1054d3e6170f59c0e3c32386a6c074e98afc8cdab"
Sep 17 15:58:58.783975 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.783866    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e948660f3344dcc04f1dc1054d3e6170f59c0e3c32386a6c074e98afc8cdab"} err="failed to get container status \"28e948660f3344dcc04f1dc1054d3e6170f59c0e3c32386a6c074e98afc8cdab\": rpc error: code = NotFound desc = could not find container \"28e948660f3344dcc04f1dc1054d3e6170f59c0e3c32386a6c074e98afc8cdab\": container with ID starting with 28e948660f3344dcc04f1dc1054d3e6170f59c0e3c32386a6c074e98afc8cdab not found: ID does not exist"
Sep 17 15:58:58.784882 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.784863    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"]
Sep 17 15:58:58.789956 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:58.789934    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-dfwmt"]
Sep 17 15:58:59.050292 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:58:59.050211    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="778a1232-40ce-4a2f-b2e6-ee8ab5389889" path="/var/lib/kubelet/pods/778a1232-40ce-4a2f-b2e6-ee8ab5389889/volumes"
Sep 17 15:59:07.100966 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:07.100923    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn" podUID="e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:59:07.971995 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:07.971958    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 15:59:12.382306 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:12.382245    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn" podUID="e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44" containerName="main" containerID="cri-o://5537106a3389cc49ac38096c3db3390576e94ebd615ed789dc244268eaa96708" gracePeriod=45
Sep 17 15:59:17.101260 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:17.101211    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn" podUID="e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:59:17.971863 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:17.971812    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 15:59:27.101187 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:27.101151    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn" podUID="e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:59:27.101684 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:27.101257    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"
Sep 17 15:59:27.971122 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:27.971082    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 15:59:37.101072 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:37.101028    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn" podUID="e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 15:59:37.972045 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:37.972009    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 15:59:42.649714 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:42.649630    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"
Sep 17 15:59:42.760529 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:42.760487    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7bh2\" (UniqueName: \"kubernetes.io/projected/e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44-kube-api-access-k7bh2\") pod \"e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44\" (UID: \"e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44\") "
Sep 17 15:59:42.760712 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:42.760596    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44-tls-certs\") pod \"e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44\" (UID: \"e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44\") "
Sep 17 15:59:42.762784 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:42.762753    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44-kube-api-access-k7bh2" (OuterVolumeSpecName: "kube-api-access-k7bh2") pod "e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44" (UID: "e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44"). InnerVolumeSpecName "kube-api-access-k7bh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:59:42.762784 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:42.762766    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44" (UID: "e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:42.861560 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:42.861517    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:59:42.861560 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:42.861555    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7bh2\" (UniqueName: \"kubernetes.io/projected/e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44-kube-api-access-k7bh2\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 15:59:42.989362 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:42.989267    2574 generic.go:356] "Generic (PLEG): container finished" podID="e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44" containerID="5537106a3389cc49ac38096c3db3390576e94ebd615ed789dc244268eaa96708" exitCode=0
Sep 17 15:59:42.989362 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:42.989331    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"
Sep 17 15:59:42.989637 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:42.989353    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn" event={"ID":"e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44","Type":"ContainerDied","Data":"5537106a3389cc49ac38096c3db3390576e94ebd615ed789dc244268eaa96708"}
Sep 17 15:59:42.989637 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:42.989468    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn" event={"ID":"e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44","Type":"ContainerDied","Data":"6f2453c2a084f6b69bbdcfe3c7b39ed1d91f7592cea1a289a23cfeea591f103c"}
Sep 17 15:59:42.989637 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:42.989505    2574 scope.go:117] "RemoveContainer" containerID="5537106a3389cc49ac38096c3db3390576e94ebd615ed789dc244268eaa96708"
Sep 17 15:59:42.998840 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:42.998821    2574 scope.go:117] "RemoveContainer" containerID="5537106a3389cc49ac38096c3db3390576e94ebd615ed789dc244268eaa96708"
Sep 17 15:59:42.999173 ip-10-0-129-37 kubenswrapper[2574]: E0917 15:59:42.999154    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5537106a3389cc49ac38096c3db3390576e94ebd615ed789dc244268eaa96708\": container with ID starting with 5537106a3389cc49ac38096c3db3390576e94ebd615ed789dc244268eaa96708 not found: ID does not exist" containerID="5537106a3389cc49ac38096c3db3390576e94ebd615ed789dc244268eaa96708"
Sep 17 15:59:42.999224 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:42.999181    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5537106a3389cc49ac38096c3db3390576e94ebd615ed789dc244268eaa96708"} err="failed to get container status \"5537106a3389cc49ac38096c3db3390576e94ebd615ed789dc244268eaa96708\": rpc error: code = NotFound desc = could not find container \"5537106a3389cc49ac38096c3db3390576e94ebd615ed789dc244268eaa96708\": container with ID starting with 5537106a3389cc49ac38096c3db3390576e94ebd615ed789dc244268eaa96708 not found: ID does not exist"
Sep 17 15:59:43.012235 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:43.012201    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"]
Sep 17 15:59:43.016897 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:43.016861    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75nmfzn"]
Sep 17 15:59:43.049999 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:43.049963    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44" path="/var/lib/kubelet/pods/e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44/volumes"
Sep 17 15:59:47.411011 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:47.410915    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:47.505907 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:47.505864    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:47.972105 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:47.972055    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 15:59:54.054254 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.054212    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:54.318977 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.318887    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"]
Sep 17 15:59:54.319388 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.319367    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44" containerName="main"
Sep 17 15:59:54.319567 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.319390    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44" containerName="main"
Sep 17 15:59:54.319567 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.319401    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778a1232-40ce-4a2f-b2e6-ee8ab5389889" containerName="llm-d-routing-sidecar"
Sep 17 15:59:54.319567 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.319409    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="778a1232-40ce-4a2f-b2e6-ee8ab5389889" containerName="llm-d-routing-sidecar"
Sep 17 15:59:54.319567 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.319420    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778a1232-40ce-4a2f-b2e6-ee8ab5389889" containerName="main"
Sep 17 15:59:54.319567 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.319430    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="778a1232-40ce-4a2f-b2e6-ee8ab5389889" containerName="main"
Sep 17 15:59:54.319871 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.319668    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4ecf6eb-c61b-4e8f-a77b-0606f1b48b44" containerName="main"
Sep 17 15:59:54.319871 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.319687    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="778a1232-40ce-4a2f-b2e6-ee8ab5389889" containerName="main"
Sep 17 15:59:54.319871 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.319701    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="778a1232-40ce-4a2f-b2e6-ee8ab5389889" containerName="llm-d-routing-sidecar"
Sep 17 15:59:54.322379 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.322354    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.324897 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.324847    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-7pqfk\""
Sep 17 15:59:54.325548 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.325525    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 15:59:54.326009 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.325630    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 15:59:54.326139 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.325757    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-7wqtb\""
Sep 17 15:59:54.326222 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.325763    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 15:59:54.334849 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.334814    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"]
Sep 17 15:59:54.378039 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.378008    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:54.473171 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.473110    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.473171 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.473178    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.473399 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.473231    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.473399 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.473320    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ece8b090-9b06-4853-8be0-c11461a52eba-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.473399 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.473360    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mp7n\" (UniqueName: \"kubernetes.io/projected/ece8b090-9b06-4853-8be0-c11461a52eba-kube-api-access-2mp7n\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.473399 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.473396    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.574597 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.574493    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ece8b090-9b06-4853-8be0-c11461a52eba-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.574597 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.574543    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mp7n\" (UniqueName: \"kubernetes.io/projected/ece8b090-9b06-4853-8be0-c11461a52eba-kube-api-access-2mp7n\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.574597 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.574575    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.574836 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.574610    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.574836 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.574635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.574836 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.574669    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.575070 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.575045    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.575122 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.575091    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.575122 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.575108    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.576986 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.576948    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.577342 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.577320    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ece8b090-9b06-4853-8be0-c11461a52eba-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.585280 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.585254    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mp7n\" (UniqueName: \"kubernetes.io/projected/ece8b090-9b06-4853-8be0-c11461a52eba-kube-api-access-2mp7n\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.635565 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.635523    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:54.653668 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.653630    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"]
Sep 17 15:59:54.656868 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.656841    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"
Sep 17 15:59:54.658909 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.658885    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-czvb6\""
Sep 17 15:59:54.666037 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.666003    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"]
Sep 17 15:59:54.775713 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.775678    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7prj\" (UniqueName: \"kubernetes.io/projected/4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b-kube-api-access-b7prj\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x\" (UID: \"4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"
Sep 17 15:59:54.775927 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.775813    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x\" (UID: \"4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"
Sep 17 15:59:54.775927 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.775832    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:54.877287 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.877254    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x\" (UID: \"4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"
Sep 17 15:59:54.877490 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.877332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7prj\" (UniqueName: \"kubernetes.io/projected/4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b-kube-api-access-b7prj\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x\" (UID: \"4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"
Sep 17 15:59:54.879867 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.879846    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x\" (UID: \"4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"
Sep 17 15:59:54.886679 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.886660    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7prj\" (UniqueName: \"kubernetes.io/projected/4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b-kube-api-access-b7prj\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x\" (UID: \"4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"
Sep 17 15:59:54.922508 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.922477    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"]
Sep 17 15:59:54.924428 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:59:54.924403    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece8b090_9b06_4853_8be0_c11461a52eba.slice/crio-30c16cc69f2cd79906033c180c7be0a30db8151dc4ac553874fe663eeed53cc2 WatchSource:0}: Error finding container 30c16cc69f2cd79906033c180c7be0a30db8151dc4ac553874fe663eeed53cc2: Status 404 returned error can't find the container with id 30c16cc69f2cd79906033c180c7be0a30db8151dc4ac553874fe663eeed53cc2
Sep 17 15:59:54.976475 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.976429    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-czvb6\""
Sep 17 15:59:54.984738 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:54.984716    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"
Sep 17 15:59:55.059405 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:55.059368    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl" event={"ID":"ece8b090-9b06-4853-8be0-c11461a52eba","Type":"ContainerStarted","Data":"894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d"}
Sep 17 15:59:55.060005 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:55.059937    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:55.060005 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:55.059975    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl" event={"ID":"ece8b090-9b06-4853-8be0-c11461a52eba","Type":"ContainerStarted","Data":"30c16cc69f2cd79906033c180c7be0a30db8151dc4ac553874fe663eeed53cc2"}
Sep 17 15:59:55.266819 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:55.266761    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"]
Sep 17 15:59:55.268914 ip-10-0-129-37 kubenswrapper[2574]: W0917 15:59:55.268879    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8a13b8_6bf1_48f5_9bcc_1a0fc859033b.slice/crio-71739bc70636d6c65ec46c02505dba20607c4072182ff0291f9e659cb043ddc7 WatchSource:0}: Error finding container 71739bc70636d6c65ec46c02505dba20607c4072182ff0291f9e659cb043ddc7: Status 404 returned error can't find the container with id 71739bc70636d6c65ec46c02505dba20607c4072182ff0291f9e659cb043ddc7
Sep 17 15:59:56.066913 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:56.066873    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl" event={"ID":"ece8b090-9b06-4853-8be0-c11461a52eba","Type":"ContainerStarted","Data":"bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335"}
Sep 17 15:59:56.068681 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:56.068656    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x" event={"ID":"4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b","Type":"ContainerStarted","Data":"c6b7530b37c1a7ab974d0403e169932b48207fa5b627f50ba624af46820e33a1"}
Sep 17 15:59:56.068817 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:56.068701    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x" event={"ID":"4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b","Type":"ContainerStarted","Data":"71739bc70636d6c65ec46c02505dba20607c4072182ff0291f9e659cb043ddc7"}
Sep 17 15:59:56.068817 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:56.068737    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"
Sep 17 15:59:56.088937 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:56.088870    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl" podStartSLOduration=2.088852628 podStartE2EDuration="2.088852628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:59:56.086103148 +0000 UTC m=+1261.557942996" watchObservedRunningTime="2026-09-17 15:59:56.088852628 +0000 UTC m=+1261.560692473"
Sep 17 15:59:56.105211 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:56.105136    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x" podStartSLOduration=2.105114253 podStartE2EDuration="2.105114253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:59:56.104844992 +0000 UTC m=+1261.576684840" watchObservedRunningTime="2026-09-17 15:59:56.105114253 +0000 UTC m=+1261.576954100"
Sep 17 15:59:56.636631 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:56.636594    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 15:59:57.971337 ip-10-0-129-37 kubenswrapper[2574]: I0917 15:59:57.971290    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 16:00:00.096584 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:00.096544    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:00.169709 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:00.169672    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:04.636472 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:04.636419    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 16:00:04.653598 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:04.653567    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 16:00:04.654638 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:04.654621    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 16:00:05.133127 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:05.133094    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 16:00:06.802505 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:06.802468    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:07.971333 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:07.971281    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 16:00:17.972070 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:17.972026    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 16:00:26.335210 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:26.335163    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:27.075957 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:27.075924    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"
Sep 17 16:00:27.971558 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:27.971505    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 16:00:29.223497 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.223424    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:29.236538 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.236500    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"]
Sep 17 16:00:29.246190 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.246160    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"]
Sep 17 16:00:29.251088 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:00:29.251048    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:00:29.251088 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:00:29.251088 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:00:29.251088 ip-10-0-129-37 kubenswrapper[2574]:  > containerID="bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335" cmd=["/bin/sleep","15"]
Sep 17 16:00:29.251432 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.251107    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl" message="" err=<
Sep 17 16:00:29.251432 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:00:29.251432 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:00:29.251432 ip-10-0-129-37 kubenswrapper[2574]:  >
Sep 17 16:00:29.251432 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:00:29.251120    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:00:29.251432 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:00:29.251432 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:00:29.251432 ip-10-0-129-37 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl" podUID="ece8b090-9b06-4853-8be0-c11461a52eba" containerName="main" containerID="cri-o://bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335"
Sep 17 16:00:29.251432 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.251174    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl" podUID="ece8b090-9b06-4853-8be0-c11461a52eba" containerName="main" containerID="cri-o://bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335" gracePeriod=60
Sep 17 16:00:29.321404 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.321347    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:29.471076 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.471047    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl" podUID="ece8b090-9b06-4853-8be0-c11461a52eba" containerName="llm-d-routing-sidecar" containerID="cri-o://894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d" gracePeriod=60
Sep 17 16:00:29.771363 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.771328    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 16:00:29.893082 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.893046    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-dshm\") pod \"ece8b090-9b06-4853-8be0-c11461a52eba\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") "
Sep 17 16:00:29.893082 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.893088    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-tmp-dir\") pod \"ece8b090-9b06-4853-8be0-c11461a52eba\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") "
Sep 17 16:00:29.893295 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.893165    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-home\") pod \"ece8b090-9b06-4853-8be0-c11461a52eba\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") "
Sep 17 16:00:29.893295 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.893245    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-model-cache\") pod \"ece8b090-9b06-4853-8be0-c11461a52eba\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") "
Sep 17 16:00:29.893295 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.893272    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ece8b090-9b06-4853-8be0-c11461a52eba-tls-certs\") pod \"ece8b090-9b06-4853-8be0-c11461a52eba\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") "
Sep 17 16:00:29.893482 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.893300    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mp7n\" (UniqueName: \"kubernetes.io/projected/ece8b090-9b06-4853-8be0-c11461a52eba-kube-api-access-2mp7n\") pod \"ece8b090-9b06-4853-8be0-c11461a52eba\" (UID: \"ece8b090-9b06-4853-8be0-c11461a52eba\") "
Sep 17 16:00:29.893482 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.893405    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ece8b090-9b06-4853-8be0-c11461a52eba" (UID: "ece8b090-9b06-4853-8be0-c11461a52eba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:29.893613 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.893469    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-home" (OuterVolumeSpecName: "home") pod "ece8b090-9b06-4853-8be0-c11461a52eba" (UID: "ece8b090-9b06-4853-8be0-c11461a52eba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:29.893613 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.893491    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-model-cache" (OuterVolumeSpecName: "model-cache") pod "ece8b090-9b06-4853-8be0-c11461a52eba" (UID: "ece8b090-9b06-4853-8be0-c11461a52eba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:29.893714 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.893626    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:00:29.893714 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.893638    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:00:29.893714 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.893647    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:00:29.895641 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.895573    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece8b090-9b06-4853-8be0-c11461a52eba-kube-api-access-2mp7n" (OuterVolumeSpecName: "kube-api-access-2mp7n") pod "ece8b090-9b06-4853-8be0-c11461a52eba" (UID: "ece8b090-9b06-4853-8be0-c11461a52eba"). InnerVolumeSpecName "kube-api-access-2mp7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:00:29.895641 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.895594    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece8b090-9b06-4853-8be0-c11461a52eba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ece8b090-9b06-4853-8be0-c11461a52eba" (UID: "ece8b090-9b06-4853-8be0-c11461a52eba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:00:29.895641 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.895602    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-dshm" (OuterVolumeSpecName: "dshm") pod "ece8b090-9b06-4853-8be0-c11461a52eba" (UID: "ece8b090-9b06-4853-8be0-c11461a52eba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:29.994243 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.994143    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ece8b090-9b06-4853-8be0-c11461a52eba-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:00:29.994243 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.994179    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mp7n\" (UniqueName: \"kubernetes.io/projected/ece8b090-9b06-4853-8be0-c11461a52eba-kube-api-access-2mp7n\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:00:29.994243 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:29.994187    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ece8b090-9b06-4853-8be0-c11461a52eba-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:00:30.255077 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.254986    2574 generic.go:356] "Generic (PLEG): container finished" podID="ece8b090-9b06-4853-8be0-c11461a52eba" containerID="bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335" exitCode=0
Sep 17 16:00:30.255077 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.255024    2574 generic.go:356] "Generic (PLEG): container finished" podID="ece8b090-9b06-4853-8be0-c11461a52eba" containerID="894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d" exitCode=0
Sep 17 16:00:30.255077 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.255050    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"
Sep 17 16:00:30.255631 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.255067    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl" event={"ID":"ece8b090-9b06-4853-8be0-c11461a52eba","Type":"ContainerDied","Data":"bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335"}
Sep 17 16:00:30.255631 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.255129    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl" event={"ID":"ece8b090-9b06-4853-8be0-c11461a52eba","Type":"ContainerDied","Data":"894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d"}
Sep 17 16:00:30.255631 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.255149    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl" event={"ID":"ece8b090-9b06-4853-8be0-c11461a52eba","Type":"ContainerDied","Data":"30c16cc69f2cd79906033c180c7be0a30db8151dc4ac553874fe663eeed53cc2"}
Sep 17 16:00:30.255631 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.255172    2574 scope.go:117] "RemoveContainer" containerID="bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335"
Sep 17 16:00:30.265269 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.265231    2574 scope.go:117] "RemoveContainer" containerID="894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d"
Sep 17 16:00:30.275742 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.275714    2574 scope.go:117] "RemoveContainer" containerID="bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335"
Sep 17 16:00:30.276121 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:00:30.276098    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335\": container with ID starting with bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335 not found: ID does not exist" containerID="bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335"
Sep 17 16:00:30.276220 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.276130    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335"} err="failed to get container status \"bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335\": rpc error: code = NotFound desc = could not find container \"bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335\": container with ID starting with bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335 not found: ID does not exist"
Sep 17 16:00:30.276220 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.276148    2574 scope.go:117] "RemoveContainer" containerID="894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d"
Sep 17 16:00:30.276498 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:00:30.276478    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d\": container with ID starting with 894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d not found: ID does not exist" containerID="894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d"
Sep 17 16:00:30.276574 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.276512    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d"} err="failed to get container status \"894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d\": rpc error: code = NotFound desc = could not find container \"894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d\": container with ID starting with 894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d not found: ID does not exist"
Sep 17 16:00:30.276574 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.276536    2574 scope.go:117] "RemoveContainer" containerID="bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335"
Sep 17 16:00:30.277328 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.277061    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335"} err="failed to get container status \"bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335\": rpc error: code = NotFound desc = could not find container \"bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335\": container with ID starting with bc520f7ef6b99b5707a682a06d3bff0b05678667c67c26a3fd166a85eb41f335 not found: ID does not exist"
Sep 17 16:00:30.277328 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.277094    2574 scope.go:117] "RemoveContainer" containerID="894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d"
Sep 17 16:00:30.277541 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.277386    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d"} err="failed to get container status \"894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d\": rpc error: code = NotFound desc = could not find container \"894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d\": container with ID starting with 894e3870efafbe8e0565b8f08f5a2ab882880728ca4c161b622ce93a3096c97d not found: ID does not exist"
Sep 17 16:00:30.279403 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.279380    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"]
Sep 17 16:00:30.283645 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:30.283616    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gqwtl"]
Sep 17 16:00:31.050055 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:31.050013    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ece8b090-9b06-4853-8be0-c11461a52eba" path="/var/lib/kubelet/pods/ece8b090-9b06-4853-8be0-c11461a52eba/volumes"
Sep 17 16:00:37.075640 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:37.075599    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x" podUID="4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:00:37.981005 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:37.980973    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 16:00:37.989239 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:37.989205    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 16:00:42.884553 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:42.884495    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:42.898105 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:42.898076    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"]
Sep 17 16:00:44.277494 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:44.277424    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x" podUID="4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b" containerName="main" containerID="cri-o://c6b7530b37c1a7ab974d0403e169932b48207fa5b627f50ba624af46820e33a1" gracePeriod=45
Sep 17 16:00:44.376173 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:44.376131    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:47.076191 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:47.076151    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x" podUID="4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:00:54.534744 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:54.534705    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:57.075252 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:57.075214    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x" podUID="4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:00:57.075674 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:57.075327    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"
Sep 17 16:00:57.922557 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:57.922508    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main" containerID="cri-o://f479c401ab78bb76db9629d86199b275b5a3ee25738b340ce680ad31e0d4fe57" gracePeriod=45
Sep 17 16:00:58.671908 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.671881    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 16:00:58.847528 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.847397    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-dshm\") pod \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") "
Sep 17 16:00:58.847528 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.847467    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-tls-certs\") pod \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") "
Sep 17 16:00:58.847528 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.847530    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-home\") pod \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") "
Sep 17 16:00:58.847811 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.847555    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-cabundle-cert\") pod \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") "
Sep 17 16:00:58.847811 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.847586    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n5t4\" (UniqueName: \"kubernetes.io/projected/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-kube-api-access-7n5t4\") pod \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") "
Sep 17 16:00:58.847811 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.847615    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-kserve-provision-location\") pod \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") "
Sep 17 16:00:58.847811 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.847672    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-model-cache\") pod \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") "
Sep 17 16:00:58.847811 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.847702    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-tmp-dir\") pod \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\" (UID: \"e24aca2e-e52c-4cc4-b5b5-245d58585b2b\") "
Sep 17 16:00:58.848100 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.847945    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e24aca2e-e52c-4cc4-b5b5-245d58585b2b" (UID: "e24aca2e-e52c-4cc4-b5b5-245d58585b2b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:00:58.848100 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.848032    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-model-cache" (OuterVolumeSpecName: "model-cache") pod "e24aca2e-e52c-4cc4-b5b5-245d58585b2b" (UID: "e24aca2e-e52c-4cc4-b5b5-245d58585b2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:58.848310 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.848283    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-home" (OuterVolumeSpecName: "home") pod "e24aca2e-e52c-4cc4-b5b5-245d58585b2b" (UID: "e24aca2e-e52c-4cc4-b5b5-245d58585b2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:58.849792 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.849763    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-dshm" (OuterVolumeSpecName: "dshm") pod "e24aca2e-e52c-4cc4-b5b5-245d58585b2b" (UID: "e24aca2e-e52c-4cc4-b5b5-245d58585b2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:58.849915 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.849871    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-kube-api-access-7n5t4" (OuterVolumeSpecName: "kube-api-access-7n5t4") pod "e24aca2e-e52c-4cc4-b5b5-245d58585b2b" (UID: "e24aca2e-e52c-4cc4-b5b5-245d58585b2b"). InnerVolumeSpecName "kube-api-access-7n5t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:00:58.850064 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.850047    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e24aca2e-e52c-4cc4-b5b5-245d58585b2b" (UID: "e24aca2e-e52c-4cc4-b5b5-245d58585b2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:00:58.866996 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.866968    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e24aca2e-e52c-4cc4-b5b5-245d58585b2b" (UID: "e24aca2e-e52c-4cc4-b5b5-245d58585b2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:58.869266 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.869244    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e24aca2e-e52c-4cc4-b5b5-245d58585b2b" (UID: "e24aca2e-e52c-4cc4-b5b5-245d58585b2b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:58.949004 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.948948    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:00:58.949004 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.948995    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:00:58.949004 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.949007    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:00:58.949004 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.949020    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:00:58.949275 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.949032    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:00:58.949275 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.949045    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n5t4\" (UniqueName: \"kubernetes.io/projected/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-kube-api-access-7n5t4\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:00:58.949275 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.949057    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:00:58.949275 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:58.949067    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e24aca2e-e52c-4cc4-b5b5-245d58585b2b-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:00:59.407267 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:59.407230    2574 generic.go:356] "Generic (PLEG): container finished" podID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerID="f479c401ab78bb76db9629d86199b275b5a3ee25738b340ce680ad31e0d4fe57" exitCode=0
Sep 17 16:00:59.407436 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:59.407286    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"
Sep 17 16:00:59.407436 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:59.407285    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" event={"ID":"e24aca2e-e52c-4cc4-b5b5-245d58585b2b","Type":"ContainerDied","Data":"f479c401ab78bb76db9629d86199b275b5a3ee25738b340ce680ad31e0d4fe57"}
Sep 17 16:00:59.407436 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:59.407351    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k" event={"ID":"e24aca2e-e52c-4cc4-b5b5-245d58585b2b","Type":"ContainerDied","Data":"78b6e21adae5ef7c44850703a7eedb30b624eb72b8ba2ab71b9f1f976cc3fe22"}
Sep 17 16:00:59.407436 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:59.407384    2574 scope.go:117] "RemoveContainer" containerID="f479c401ab78bb76db9629d86199b275b5a3ee25738b340ce680ad31e0d4fe57"
Sep 17 16:00:59.416057 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:59.416037    2574 scope.go:117] "RemoveContainer" containerID="6ec24b30ea9a4b1c0f85095325f39fae2c15d97f390f90f33258a0dcb0e2bd38"
Sep 17 16:00:59.423795 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:59.423774    2574 scope.go:117] "RemoveContainer" containerID="f479c401ab78bb76db9629d86199b275b5a3ee25738b340ce680ad31e0d4fe57"
Sep 17 16:00:59.424179 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:00:59.424147    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f479c401ab78bb76db9629d86199b275b5a3ee25738b340ce680ad31e0d4fe57\": container with ID starting with f479c401ab78bb76db9629d86199b275b5a3ee25738b340ce680ad31e0d4fe57 not found: ID does not exist" containerID="f479c401ab78bb76db9629d86199b275b5a3ee25738b340ce680ad31e0d4fe57"
Sep 17 16:00:59.424273 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:59.424178    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f479c401ab78bb76db9629d86199b275b5a3ee25738b340ce680ad31e0d4fe57"} err="failed to get container status \"f479c401ab78bb76db9629d86199b275b5a3ee25738b340ce680ad31e0d4fe57\": rpc error: code = NotFound desc = could not find container \"f479c401ab78bb76db9629d86199b275b5a3ee25738b340ce680ad31e0d4fe57\": container with ID starting with f479c401ab78bb76db9629d86199b275b5a3ee25738b340ce680ad31e0d4fe57 not found: ID does not exist"
Sep 17 16:00:59.424273 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:59.424201    2574 scope.go:117] "RemoveContainer" containerID="6ec24b30ea9a4b1c0f85095325f39fae2c15d97f390f90f33258a0dcb0e2bd38"
Sep 17 16:00:59.424638 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:00:59.424605    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec24b30ea9a4b1c0f85095325f39fae2c15d97f390f90f33258a0dcb0e2bd38\": container with ID starting with 6ec24b30ea9a4b1c0f85095325f39fae2c15d97f390f90f33258a0dcb0e2bd38 not found: ID does not exist" containerID="6ec24b30ea9a4b1c0f85095325f39fae2c15d97f390f90f33258a0dcb0e2bd38"
Sep 17 16:00:59.424754 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:59.424647    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec24b30ea9a4b1c0f85095325f39fae2c15d97f390f90f33258a0dcb0e2bd38"} err="failed to get container status \"6ec24b30ea9a4b1c0f85095325f39fae2c15d97f390f90f33258a0dcb0e2bd38\": rpc error: code = NotFound desc = could not find container \"6ec24b30ea9a4b1c0f85095325f39fae2c15d97f390f90f33258a0dcb0e2bd38\": container with ID starting with 6ec24b30ea9a4b1c0f85095325f39fae2c15d97f390f90f33258a0dcb0e2bd38 not found: ID does not exist"
Sep 17 16:00:59.425840 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:59.425818    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"]
Sep 17 16:00:59.432271 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:00:59.432248    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-656ddb986c-l7h4k"]
Sep 17 16:01:01.049402 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:01.049367    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" path="/var/lib/kubelet/pods/e24aca2e-e52c-4cc4-b5b5-245d58585b2b/volumes"
Sep 17 16:01:07.075668 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:07.075628    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x" podUID="4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:01:14.487677 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:14.487639    2574 generic.go:356] "Generic (PLEG): container finished" podID="4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b" containerID="c6b7530b37c1a7ab974d0403e169932b48207fa5b627f50ba624af46820e33a1" exitCode=0
Sep 17 16:01:14.488033 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:14.487716    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x" event={"ID":"4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b","Type":"ContainerDied","Data":"c6b7530b37c1a7ab974d0403e169932b48207fa5b627f50ba624af46820e33a1"}
Sep 17 16:01:14.530428 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:14.530401    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"
Sep 17 16:01:14.578683 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:14.578603    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b-tls-certs\") pod \"4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b\" (UID: \"4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b\") "
Sep 17 16:01:14.578835 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:14.578681    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7prj\" (UniqueName: \"kubernetes.io/projected/4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b-kube-api-access-b7prj\") pod \"4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b\" (UID: \"4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b\") "
Sep 17 16:01:14.580754 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:14.580720    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b" (UID: "4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:01:14.580870 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:14.580774    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b-kube-api-access-b7prj" (OuterVolumeSpecName: "kube-api-access-b7prj") pod "4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b" (UID: "4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b"). InnerVolumeSpecName "kube-api-access-b7prj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:01:14.680154 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:14.680117    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7prj\" (UniqueName: \"kubernetes.io/projected/4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b-kube-api-access-b7prj\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:01:14.680154 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:14.680148    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:01:15.493735 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:15.493704    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"
Sep 17 16:01:15.493735 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:15.493719    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x" event={"ID":"4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b","Type":"ContainerDied","Data":"71739bc70636d6c65ec46c02505dba20607c4072182ff0291f9e659cb043ddc7"}
Sep 17 16:01:15.494226 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:15.493777    2574 scope.go:117] "RemoveContainer" containerID="c6b7530b37c1a7ab974d0403e169932b48207fa5b627f50ba624af46820e33a1"
Sep 17 16:01:15.531374 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:15.531339    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"]
Sep 17 16:01:15.538015 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:15.537986    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedb2w5x"]
Sep 17 16:01:17.048588 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:17.048549    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b" path="/var/lib/kubelet/pods/4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b/volumes"
Sep 17 16:01:19.310269 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:19.310221    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:19.435466 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:19.435419    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:22.803391 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803354    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"]
Sep 17 16:01:22.803785 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803737    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="storage-initializer"
Sep 17 16:01:22.803785 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803750    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="storage-initializer"
Sep 17 16:01:22.803785 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803765    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece8b090-9b06-4853-8be0-c11461a52eba" containerName="llm-d-routing-sidecar"
Sep 17 16:01:22.803785 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803770    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece8b090-9b06-4853-8be0-c11461a52eba" containerName="llm-d-routing-sidecar"
Sep 17 16:01:22.803785 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803778    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece8b090-9b06-4853-8be0-c11461a52eba" containerName="main"
Sep 17 16:01:22.803785 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803784    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece8b090-9b06-4853-8be0-c11461a52eba" containerName="main"
Sep 17 16:01:22.803996 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803795    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main"
Sep 17 16:01:22.803996 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803801    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main"
Sep 17 16:01:22.803996 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803809    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b" containerName="main"
Sep 17 16:01:22.803996 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803814    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b" containerName="main"
Sep 17 16:01:22.803996 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803871    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece8b090-9b06-4853-8be0-c11461a52eba" containerName="main"
Sep 17 16:01:22.803996 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803879    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e24aca2e-e52c-4cc4-b5b5-245d58585b2b" containerName="main"
Sep 17 16:01:22.803996 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803884    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece8b090-9b06-4853-8be0-c11461a52eba" containerName="llm-d-routing-sidecar"
Sep 17 16:01:22.803996 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.803889    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b8a13b8-6bf1-48f5-9bcc-1a0fc859033b" containerName="main"
Sep 17 16:01:22.808492 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.808468    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.810854 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.810827    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:01:22.810982 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.810885    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-jrgx7\""
Sep 17 16:01:22.811120 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.811105    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 16:01:22.811899 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.811874    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 16:01:22.812011 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.811988    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 16:01:22.812072 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.811992    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 16:01:22.821287 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.821262    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"]
Sep 17 16:01:22.847261 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.847231    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-home\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.847418 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.847270    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-dshm\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.847418 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.847293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0235fe02-e19e-4010-a205-214c0c5ea118-tls-certs\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.847418 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.847386    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-model-cache\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.847778 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.847465    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wldd8\" (UniqueName: \"kubernetes.io/projected/0235fe02-e19e-4010-a205-214c0c5ea118-kube-api-access-wldd8\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.847778 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.847504    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.847778 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.847558    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.847778 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.847629    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0235fe02-e19e-4010-a205-214c0c5ea118-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.849115 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.849092    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:22.948036 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.947997    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-model-cache\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.948218 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.948047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wldd8\" (UniqueName: \"kubernetes.io/projected/0235fe02-e19e-4010-a205-214c0c5ea118-kube-api-access-wldd8\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.948218 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.948069    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.948218 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.948099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.948218 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.948134    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0235fe02-e19e-4010-a205-214c0c5ea118-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.948218 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.948153    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-home\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.948218 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.948180    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-dshm\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.948218 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.948206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0235fe02-e19e-4010-a205-214c0c5ea118-tls-certs\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.948600 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.948569    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.948682 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.948611    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-model-cache\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.948682 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.948671    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-home\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.948792 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.948719    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.949045 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.949025    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0235fe02-e19e-4010-a205-214c0c5ea118-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.950472 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.950433    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-dshm\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.950782 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.950762    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0235fe02-e19e-4010-a205-214c0c5ea118-tls-certs\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:22.956390 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:22.956366    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wldd8\" (UniqueName: \"kubernetes.io/projected/0235fe02-e19e-4010-a205-214c0c5ea118-kube-api-access-wldd8\") pod \"tokenizer-clean-path-test-kserve-644886d9cf-df2lv\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:23.041136 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.041096    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"]
Sep 17 16:01:23.046891 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.046857    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:01:23.056502 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.056080    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"]
Sep 17 16:01:23.082466 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.082416    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:23.119715 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.119676    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:23.146977 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.146948    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:23.150941 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.150918    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847e4d63-dace-4185-a0ff-fc32744b887a-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:01:23.151040 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.150963    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847e4d63-dace-4185-a0ff-fc32744b887a-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:01:23.151579 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.151459    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847e4d63-dace-4185-a0ff-fc32744b887a-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:01:23.151579 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.151527    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/847e4d63-dace-4185-a0ff-fc32744b887a-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:01:23.252103 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.252064    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847e4d63-dace-4185-a0ff-fc32744b887a-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:01:23.252289 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.252116    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/847e4d63-dace-4185-a0ff-fc32744b887a-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:01:23.252289 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.252164    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847e4d63-dace-4185-a0ff-fc32744b887a-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:01:23.252420 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.252311    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847e4d63-dace-4185-a0ff-fc32744b887a-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:01:23.252542 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.252486    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847e4d63-dace-4185-a0ff-fc32744b887a-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:01:23.252607 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.252557    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847e4d63-dace-4185-a0ff-fc32744b887a-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:01:23.252829 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.252803    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/847e4d63-dace-4185-a0ff-fc32744b887a-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:01:23.254930 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.254901    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847e4d63-dace-4185-a0ff-fc32744b887a-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:01:23.360662 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.360623    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:01:23.401985 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.401857    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"]
Sep 17 16:01:23.404274 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:01:23.404246    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0235fe02_e19e_4010_a205_214c0c5ea118.slice/crio-897c1c4610ef0cc6e87c6eda2b290f183dafd6477a92ef8045bca0c457015d4e WatchSource:0}: Error finding container 897c1c4610ef0cc6e87c6eda2b290f183dafd6477a92ef8045bca0c457015d4e: Status 404 returned error can't find the container with id 897c1c4610ef0cc6e87c6eda2b290f183dafd6477a92ef8045bca0c457015d4e
Sep 17 16:01:23.534243 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.534080    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv" event={"ID":"0235fe02-e19e-4010-a205-214c0c5ea118","Type":"ContainerStarted","Data":"44e5ef0451ce266ee792570f85e6650d281ce2b895420901acde811945306c23"}
Sep 17 16:01:23.534243 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.534153    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv" event={"ID":"0235fe02-e19e-4010-a205-214c0c5ea118","Type":"ContainerStarted","Data":"897c1c4610ef0cc6e87c6eda2b290f183dafd6477a92ef8045bca0c457015d4e"}
Sep 17 16:01:23.643995 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:23.643963    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"]
Sep 17 16:01:23.647066 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:01:23.647037    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847e4d63_dace_4185_a0ff_fc32744b887a.slice/crio-69ca883d985d641fa4436d86dc39e7a45b5dfb7f4bec05dfe8a6a6180bdb2e58 WatchSource:0}: Error finding container 69ca883d985d641fa4436d86dc39e7a45b5dfb7f4bec05dfe8a6a6180bdb2e58: Status 404 returned error can't find the container with id 69ca883d985d641fa4436d86dc39e7a45b5dfb7f4bec05dfe8a6a6180bdb2e58
Sep 17 16:01:24.540412 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:24.540366    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb" event={"ID":"847e4d63-dace-4185-a0ff-fc32744b887a","Type":"ContainerStarted","Data":"271aadb8c759eb80261e78e25c99486e6656bde8d7e7c3ee3910b8fdce29baaf"}
Sep 17 16:01:24.540934 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:24.540422    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb" event={"ID":"847e4d63-dace-4185-a0ff-fc32744b887a","Type":"ContainerStarted","Data":"69ca883d985d641fa4436d86dc39e7a45b5dfb7f4bec05dfe8a6a6180bdb2e58"}
Sep 17 16:01:26.555349 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:26.555306    2574 generic.go:356] "Generic (PLEG): container finished" podID="847e4d63-dace-4185-a0ff-fc32744b887a" containerID="271aadb8c759eb80261e78e25c99486e6656bde8d7e7c3ee3910b8fdce29baaf" exitCode=0
Sep 17 16:01:26.555784 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:26.555373    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb" event={"ID":"847e4d63-dace-4185-a0ff-fc32744b887a","Type":"ContainerDied","Data":"271aadb8c759eb80261e78e25c99486e6656bde8d7e7c3ee3910b8fdce29baaf"}
Sep 17 16:01:26.557349 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:26.557324    2574 generic.go:356] "Generic (PLEG): container finished" podID="0235fe02-e19e-4010-a205-214c0c5ea118" containerID="44e5ef0451ce266ee792570f85e6650d281ce2b895420901acde811945306c23" exitCode=0
Sep 17 16:01:26.557501 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:26.557376    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv" event={"ID":"0235fe02-e19e-4010-a205-214c0c5ea118","Type":"ContainerDied","Data":"44e5ef0451ce266ee792570f85e6650d281ce2b895420901acde811945306c23"}
Sep 17 16:01:26.631733 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:26.631702    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:27.563737 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:27.563705    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv" event={"ID":"0235fe02-e19e-4010-a205-214c0c5ea118","Type":"ContainerStarted","Data":"cdbcd1e6caecc2f98d20ff79e3e2075a5307b66ef3bd9bb2d1adc8fcd43e9319"}
Sep 17 16:01:27.585522 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:27.585466    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv" podStartSLOduration=2.96622584 podStartE2EDuration="5.585427192s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.619201352s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:27.583190857 +0000 UTC m=+1353.055030701" watchObservedRunningTime="2026-09-17 16:01:27.585427192 +0000 UTC m=+1353.057267035"
Sep 17 16:01:28.120624 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:28.120584    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:32.868673 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:32.868636    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:33.005255 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:33.005202    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:33.117266 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:33.117200    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:33.120351 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:33.120276    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:33.138643 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:33.138565    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:33.628666 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:33.628632    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:01:37.098099 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:37.098063    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:39.521991 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:39.521947    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:46.049051 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:46.049011    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:46.161052 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:46.161010    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:58.649735 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:58.649682    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:58.900254 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:01:58.900174    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:14.643008 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:14.642968    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:14.859498 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:14.859457    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb" event={"ID":"847e4d63-dace-4185-a0ff-fc32744b887a","Type":"ContainerStarted","Data":"6ba6a49184a0288d70caef5d668ae41e5095ff15d068bf3a9673bc1ce88e0e2f"}
Sep 17 16:02:14.859706 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:14.859684    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:02:14.885118 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:14.885067    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb" podStartSLOduration=1.9343027510000002 podStartE2EDuration="51.885051236s" totalImagesPullingTime="47.334915364s" totalInitContainerRuntime="2.615833121s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:02:14.883569642 +0000 UTC m=+1400.355409501" watchObservedRunningTime="2026-09-17 16:02:14.885051236 +0000 UTC m=+1400.356891081"
Sep 17 16:02:23.259332 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:23.259284    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:45.876496 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:45.876465    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:02:47.274651 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:47.274575    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb_847e4d63-dace-4185-a0ff-fc32744b887a/main/0.log"
Sep 17 16:02:47.519724 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:47.519687    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:47.542739 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:47.542648    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"]
Sep 17 16:02:47.552857 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:02:47.552812    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:02:47.552857 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:02:47.552857 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:02:47.552857 ip-10-0-129-37 kubenswrapper[2574]:  > containerID="cdbcd1e6caecc2f98d20ff79e3e2075a5307b66ef3bd9bb2d1adc8fcd43e9319" cmd=["/bin/sleep","15"]
Sep 17 16:02:47.553215 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:47.552881    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv" message="" err=<
Sep 17 16:02:47.553215 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:02:47.553215 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:02:47.553215 ip-10-0-129-37 kubenswrapper[2574]:  >
Sep 17 16:02:47.553215 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:02:47.552895    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:02:47.553215 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:02:47.553215 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:02:47.553215 ip-10-0-129-37 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv" podUID="0235fe02-e19e-4010-a205-214c0c5ea118" containerName="main" containerID="cri-o://cdbcd1e6caecc2f98d20ff79e3e2075a5307b66ef3bd9bb2d1adc8fcd43e9319"
Sep 17 16:02:47.553215 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:47.552940    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv" podUID="0235fe02-e19e-4010-a205-214c0c5ea118" containerName="main" containerID="cri-o://cdbcd1e6caecc2f98d20ff79e3e2075a5307b66ef3bd9bb2d1adc8fcd43e9319" gracePeriod=60
Sep 17 16:02:47.556141 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:47.556113    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"]
Sep 17 16:02:47.556377 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:47.556350    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb" podUID="847e4d63-dace-4185-a0ff-fc32744b887a" containerName="main" containerID="cri-o://6ba6a49184a0288d70caef5d668ae41e5095ff15d068bf3a9673bc1ce88e0e2f" gracePeriod=30
Sep 17 16:02:47.612661 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:47.612618    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv" podUID="0235fe02-e19e-4010-a205-214c0c5ea118" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 16:02:47.639910 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:47.639875    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:48.031416 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.031382    2574 generic.go:356] "Generic (PLEG): container finished" podID="0235fe02-e19e-4010-a205-214c0c5ea118" containerID="cdbcd1e6caecc2f98d20ff79e3e2075a5307b66ef3bd9bb2d1adc8fcd43e9319" exitCode=0
Sep 17 16:02:48.031600 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.031482    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv" event={"ID":"0235fe02-e19e-4010-a205-214c0c5ea118","Type":"ContainerDied","Data":"cdbcd1e6caecc2f98d20ff79e3e2075a5307b66ef3bd9bb2d1adc8fcd43e9319"}
Sep 17 16:02:48.033572 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.033550    2574 generic.go:356] "Generic (PLEG): container finished" podID="847e4d63-dace-4185-a0ff-fc32744b887a" containerID="6ba6a49184a0288d70caef5d668ae41e5095ff15d068bf3a9673bc1ce88e0e2f" exitCode=0
Sep 17 16:02:48.033708 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.033585    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb" event={"ID":"847e4d63-dace-4185-a0ff-fc32744b887a","Type":"ContainerDied","Data":"6ba6a49184a0288d70caef5d668ae41e5095ff15d068bf3a9673bc1ce88e0e2f"}
Sep 17 16:02:48.202932 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.202903    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:02:48.272790 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.272761    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/847e4d63-dace-4185-a0ff-fc32744b887a-cabundle-cert\") pod \"847e4d63-dace-4185-a0ff-fc32744b887a\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") "
Sep 17 16:02:48.272980 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.272809    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847e4d63-dace-4185-a0ff-fc32744b887a-tmp-dir\") pod \"847e4d63-dace-4185-a0ff-fc32744b887a\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") "
Sep 17 16:02:48.272980 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.272826    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847e4d63-dace-4185-a0ff-fc32744b887a-tls-certs\") pod \"847e4d63-dace-4185-a0ff-fc32744b887a\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") "
Sep 17 16:02:48.272980 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.272844    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847e4d63-dace-4185-a0ff-fc32744b887a-kserve-provision-location\") pod \"847e4d63-dace-4185-a0ff-fc32744b887a\" (UID: \"847e4d63-dace-4185-a0ff-fc32744b887a\") "
Sep 17 16:02:48.273217 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.273189    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847e4d63-dace-4185-a0ff-fc32744b887a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "847e4d63-dace-4185-a0ff-fc32744b887a" (UID: "847e4d63-dace-4185-a0ff-fc32744b887a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:02:48.273281 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.273243    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847e4d63-dace-4185-a0ff-fc32744b887a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "847e4d63-dace-4185-a0ff-fc32744b887a" (UID: "847e4d63-dace-4185-a0ff-fc32744b887a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:02:48.275062 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.275044    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847e4d63-dace-4185-a0ff-fc32744b887a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "847e4d63-dace-4185-a0ff-fc32744b887a" (UID: "847e4d63-dace-4185-a0ff-fc32744b887a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:02:48.295324 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.295291    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847e4d63-dace-4185-a0ff-fc32744b887a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "847e4d63-dace-4185-a0ff-fc32744b887a" (UID: "847e4d63-dace-4185-a0ff-fc32744b887a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:02:48.374335 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.374299    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847e4d63-dace-4185-a0ff-fc32744b887a-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.374335 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.374327    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847e4d63-dace-4185-a0ff-fc32744b887a-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.374335 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.374337    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/847e4d63-dace-4185-a0ff-fc32744b887a-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.374588 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.374346    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/847e4d63-dace-4185-a0ff-fc32744b887a-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.441038 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.440958    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:02:48.475317 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.475289    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-dshm\") pod \"0235fe02-e19e-4010-a205-214c0c5ea118\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") "
Sep 17 16:02:48.475553 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.475354    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-kserve-provision-location\") pod \"0235fe02-e19e-4010-a205-214c0c5ea118\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") "
Sep 17 16:02:48.475553 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.475392    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0235fe02-e19e-4010-a205-214c0c5ea118-cabundle-cert\") pod \"0235fe02-e19e-4010-a205-214c0c5ea118\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") "
Sep 17 16:02:48.475553 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.475409    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0235fe02-e19e-4010-a205-214c0c5ea118-tls-certs\") pod \"0235fe02-e19e-4010-a205-214c0c5ea118\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") "
Sep 17 16:02:48.475553 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.475456    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-home\") pod \"0235fe02-e19e-4010-a205-214c0c5ea118\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") "
Sep 17 16:02:48.475553 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.475491    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wldd8\" (UniqueName: \"kubernetes.io/projected/0235fe02-e19e-4010-a205-214c0c5ea118-kube-api-access-wldd8\") pod \"0235fe02-e19e-4010-a205-214c0c5ea118\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") "
Sep 17 16:02:48.475553 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.475541    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-tmp-dir\") pod \"0235fe02-e19e-4010-a205-214c0c5ea118\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") "
Sep 17 16:02:48.475893 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.475577    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-model-cache\") pod \"0235fe02-e19e-4010-a205-214c0c5ea118\" (UID: \"0235fe02-e19e-4010-a205-214c0c5ea118\") "
Sep 17 16:02:48.475893 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.475780    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0235fe02-e19e-4010-a205-214c0c5ea118-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0235fe02-e19e-4010-a205-214c0c5ea118" (UID: "0235fe02-e19e-4010-a205-214c0c5ea118"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:02:48.476013 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.475946    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-home" (OuterVolumeSpecName: "home") pod "0235fe02-e19e-4010-a205-214c0c5ea118" (UID: "0235fe02-e19e-4010-a205-214c0c5ea118"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:02:48.476013 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.475960    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0235fe02-e19e-4010-a205-214c0c5ea118" (UID: "0235fe02-e19e-4010-a205-214c0c5ea118"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:02:48.476121 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.476032    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-model-cache" (OuterVolumeSpecName: "model-cache") pod "0235fe02-e19e-4010-a205-214c0c5ea118" (UID: "0235fe02-e19e-4010-a205-214c0c5ea118"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:02:48.477898 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.477865    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0235fe02-e19e-4010-a205-214c0c5ea118-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0235fe02-e19e-4010-a205-214c0c5ea118" (UID: "0235fe02-e19e-4010-a205-214c0c5ea118"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:02:48.478026 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.478009    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-dshm" (OuterVolumeSpecName: "dshm") pod "0235fe02-e19e-4010-a205-214c0c5ea118" (UID: "0235fe02-e19e-4010-a205-214c0c5ea118"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:02:48.478339 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.478317    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0235fe02-e19e-4010-a205-214c0c5ea118-kube-api-access-wldd8" (OuterVolumeSpecName: "kube-api-access-wldd8") pod "0235fe02-e19e-4010-a205-214c0c5ea118" (UID: "0235fe02-e19e-4010-a205-214c0c5ea118"). InnerVolumeSpecName "kube-api-access-wldd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:02:48.497416 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.497384    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0235fe02-e19e-4010-a205-214c0c5ea118" (UID: "0235fe02-e19e-4010-a205-214c0c5ea118"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:02:48.576836 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.576793    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.576836 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.576836    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wldd8\" (UniqueName: \"kubernetes.io/projected/0235fe02-e19e-4010-a205-214c0c5ea118-kube-api-access-wldd8\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.577020 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.576849    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.577020 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.576862    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.577020 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.576873    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.577020 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.576883    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0235fe02-e19e-4010-a205-214c0c5ea118-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.577020 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.576894    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0235fe02-e19e-4010-a205-214c0c5ea118-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.577020 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:48.576907    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0235fe02-e19e-4010-a205-214c0c5ea118-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:02:49.040525 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:49.040430    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"
Sep 17 16:02:49.040525 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:49.040469    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb" event={"ID":"847e4d63-dace-4185-a0ff-fc32744b887a","Type":"ContainerDied","Data":"69ca883d985d641fa4436d86dc39e7a45b5dfb7f4bec05dfe8a6a6180bdb2e58"}
Sep 17 16:02:49.040752 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:49.040544    2574 scope.go:117] "RemoveContainer" containerID="6ba6a49184a0288d70caef5d668ae41e5095ff15d068bf3a9673bc1ce88e0e2f"
Sep 17 16:02:49.042388 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:49.042366    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"
Sep 17 16:02:49.053412 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:49.053255    2574 scope.go:117] "RemoveContainer" containerID="271aadb8c759eb80261e78e25c99486e6656bde8d7e7c3ee3910b8fdce29baaf"
Sep 17 16:02:49.057381 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:49.055306    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv" event={"ID":"0235fe02-e19e-4010-a205-214c0c5ea118","Type":"ContainerDied","Data":"897c1c4610ef0cc6e87c6eda2b290f183dafd6477a92ef8045bca0c457015d4e"}
Sep 17 16:02:49.066350 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:49.066328    2574 scope.go:117] "RemoveContainer" containerID="cdbcd1e6caecc2f98d20ff79e3e2075a5307b66ef3bd9bb2d1adc8fcd43e9319"
Sep 17 16:02:49.070369 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:49.070347    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"]
Sep 17 16:02:49.075150 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:49.075128    2574 scope.go:117] "RemoveContainer" containerID="44e5ef0451ce266ee792570f85e6650d281ce2b895420901acde811945306c23"
Sep 17 16:02:49.076570 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:49.076548    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6d47bbdb94-bplgb"]
Sep 17 16:02:49.090261 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:49.090236    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"]
Sep 17 16:02:49.095165 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:49.095141    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-644886d9cf-df2lv"]
Sep 17 16:02:50.171933 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:50.171902    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:50.231128 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:50.231085    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:51.048880 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:51.048846    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0235fe02-e19e-4010-a205-214c0c5ea118" path="/var/lib/kubelet/pods/0235fe02-e19e-4010-a205-214c0c5ea118/volumes"
Sep 17 16:02:51.049313 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:02:51.049299    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="847e4d63-dace-4185-a0ff-fc32744b887a" path="/var/lib/kubelet/pods/847e4d63-dace-4185-a0ff-fc32744b887a/volumes"
Sep 17 16:03:21.626304 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:21.626266    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:37.611998 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:37.611958    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:37.692902 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:37.692865    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:44.509336 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:44.509301    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:50.480765 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:50.480697    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:50.541465 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:50.541390    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:51.510880 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.510845    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"]
Sep 17 16:03:51.511360 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.511328    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0235fe02-e19e-4010-a205-214c0c5ea118" containerName="main"
Sep 17 16:03:51.511360 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.511352    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0235fe02-e19e-4010-a205-214c0c5ea118" containerName="main"
Sep 17 16:03:51.511486 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.511392    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0235fe02-e19e-4010-a205-214c0c5ea118" containerName="storage-initializer"
Sep 17 16:03:51.511486 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.511402    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0235fe02-e19e-4010-a205-214c0c5ea118" containerName="storage-initializer"
Sep 17 16:03:51.511486 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.511411    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847e4d63-dace-4185-a0ff-fc32744b887a" containerName="storage-initializer"
Sep 17 16:03:51.511486 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.511419    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="847e4d63-dace-4185-a0ff-fc32744b887a" containerName="storage-initializer"
Sep 17 16:03:51.511486 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.511426    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847e4d63-dace-4185-a0ff-fc32744b887a" containerName="main"
Sep 17 16:03:51.511486 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.511432    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="847e4d63-dace-4185-a0ff-fc32744b887a" containerName="main"
Sep 17 16:03:51.511708 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.511527    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="847e4d63-dace-4185-a0ff-fc32744b887a" containerName="main"
Sep 17 16:03:51.511708 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.511539    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0235fe02-e19e-4010-a205-214c0c5ea118" containerName="main"
Sep 17 16:03:51.513803 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.513783    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.516028 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.516004    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:03:51.516142 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.516119    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-j7q5k\""
Sep 17 16:03:51.516662 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.516639    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 16:03:51.516756 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.516663    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 16:03:51.516756 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.516663    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 16:03:51.516756 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.516685    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 16:03:51.528591 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.528560    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"]
Sep 17 16:03:51.545429 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.545395    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:51.610299 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.610262    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.610299 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.610302    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.610551 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.610345    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w7lt\" (UniqueName: \"kubernetes.io/projected/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-kube-api-access-6w7lt\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.610551 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.610387    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-dshm\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.610551 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.610407    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.610551 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.610424    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.610551 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.610468    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-model-cache\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.610551 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.610517    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-home\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.711409 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.711367    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w7lt\" (UniqueName: \"kubernetes.io/projected/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-kube-api-access-6w7lt\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.711596 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.711424    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-dshm\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.711596 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.711490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.711596 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.711517    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.711726 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.711653    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-model-cache\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.711807 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.711725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-home\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.711807 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.711783    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.711884 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.711813    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.711955 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.711929    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.712049 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.712038    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-model-cache\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.712138 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.712122    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.712188 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.712131    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-home\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.712520 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.712501    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.714428 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.714406    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-dshm\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.714662 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.714648    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.722219 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.722197    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w7lt\" (UniqueName: \"kubernetes.io/projected/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-kube-api-access-6w7lt\") pod \"tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.762781 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.762700    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"]
Sep 17 16:03:51.766364 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.766341    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:51.780275 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.780240    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"]
Sep 17 16:03:51.803226 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.803186    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:51.812843 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.812806    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6557a20d-459b-4587-88e7-1d20bd3a0243-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:51.813023 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.812868    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6557a20d-459b-4587-88e7-1d20bd3a0243-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:51.813023 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.812901    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6557a20d-459b-4587-88e7-1d20bd3a0243-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:51.813162 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.813059    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6557a20d-459b-4587-88e7-1d20bd3a0243-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:51.825474 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.825429    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:03:51.880918 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.880885    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:51.913880 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.913849    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6557a20d-459b-4587-88e7-1d20bd3a0243-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:51.914043 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.913891    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6557a20d-459b-4587-88e7-1d20bd3a0243-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:51.914043 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.913959    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6557a20d-459b-4587-88e7-1d20bd3a0243-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:51.914043 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.913987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6557a20d-459b-4587-88e7-1d20bd3a0243-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:51.914353 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.914325    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6557a20d-459b-4587-88e7-1d20bd3a0243-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:51.914428 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.914354    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6557a20d-459b-4587-88e7-1d20bd3a0243-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:51.914644 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.914627    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6557a20d-459b-4587-88e7-1d20bd3a0243-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:51.916375 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:51.916358    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6557a20d-459b-4587-88e7-1d20bd3a0243-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:52.076679 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:52.076594    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:52.101666 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:52.101642    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"]
Sep 17 16:03:52.103521 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:03:52.103481    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95991c50_b4f9_40c8_8a12_9a7b4a6b4ac5.slice/crio-402c6706bb444d22a01569e11fd3e983559eb75bef9571e8cb35ef347af93767 WatchSource:0}: Error finding container 402c6706bb444d22a01569e11fd3e983559eb75bef9571e8cb35ef347af93767: Status 404 returned error can't find the container with id 402c6706bb444d22a01569e11fd3e983559eb75bef9571e8cb35ef347af93767
Sep 17 16:03:52.346669 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:52.346587    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj" event={"ID":"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5","Type":"ContainerStarted","Data":"4a3c5249369bdc806b3e330685db35785b8d351773e29719f23def8cdd6dab41"}
Sep 17 16:03:52.346669 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:52.346639    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj" event={"ID":"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5","Type":"ContainerStarted","Data":"402c6706bb444d22a01569e11fd3e983559eb75bef9571e8cb35ef347af93767"}
Sep 17 16:03:52.358960 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:52.358928    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"]
Sep 17 16:03:52.359614 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:03:52.359577    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6557a20d_459b_4587_88e7_1d20bd3a0243.slice/crio-72f5cadd1e9c869e993853990d2156c1a0bc03cb013d833b0206f3527048aab8 WatchSource:0}: Error finding container 72f5cadd1e9c869e993853990d2156c1a0bc03cb013d833b0206f3527048aab8: Status 404 returned error can't find the container with id 72f5cadd1e9c869e993853990d2156c1a0bc03cb013d833b0206f3527048aab8
Sep 17 16:03:53.354157 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:53.354121    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh" event={"ID":"6557a20d-459b-4587-88e7-1d20bd3a0243","Type":"ContainerStarted","Data":"33aa1c7de3a6433a0fe84d068e8e4ac71b002fd66056d153205e000f26ea35d7"}
Sep 17 16:03:53.354649 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:53.354176    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh" event={"ID":"6557a20d-459b-4587-88e7-1d20bd3a0243","Type":"ContainerStarted","Data":"72f5cadd1e9c869e993853990d2156c1a0bc03cb013d833b0206f3527048aab8"}
Sep 17 16:03:55.370658 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:55.370621    2574 generic.go:356] "Generic (PLEG): container finished" podID="6557a20d-459b-4587-88e7-1d20bd3a0243" containerID="33aa1c7de3a6433a0fe84d068e8e4ac71b002fd66056d153205e000f26ea35d7" exitCode=0
Sep 17 16:03:55.371087 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:55.370697    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh" event={"ID":"6557a20d-459b-4587-88e7-1d20bd3a0243","Type":"ContainerDied","Data":"33aa1c7de3a6433a0fe84d068e8e4ac71b002fd66056d153205e000f26ea35d7"}
Sep 17 16:03:55.372223 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:55.372203    2574 generic.go:356] "Generic (PLEG): container finished" podID="95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" containerID="4a3c5249369bdc806b3e330685db35785b8d351773e29719f23def8cdd6dab41" exitCode=0
Sep 17 16:03:55.372298 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:55.372269    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj" event={"ID":"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5","Type":"ContainerDied","Data":"4a3c5249369bdc806b3e330685db35785b8d351773e29719f23def8cdd6dab41"}
Sep 17 16:03:56.379504 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:56.379459    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh" event={"ID":"6557a20d-459b-4587-88e7-1d20bd3a0243","Type":"ContainerStarted","Data":"c62e12557ed08cb28c9faeb35449f0892e5ab4e92ca732b2c4e9a3df9fcbe2f4"}
Sep 17 16:03:56.379970 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:56.379732    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:03:56.381403 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:56.381375    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj" event={"ID":"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5","Type":"ContainerStarted","Data":"c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7"}
Sep 17 16:03:56.400680 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:56.400631    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh" podStartSLOduration=2.813351912 podStartE2EDuration="5.400611151s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.587259239s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:03:56.399991493 +0000 UTC m=+1501.871831340" watchObservedRunningTime="2026-09-17 16:03:56.400611151 +0000 UTC m=+1501.872451001"
Sep 17 16:03:56.421915 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:56.421865    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj" podStartSLOduration=2.788347719 podStartE2EDuration="5.421848974s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.633501255s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:03:56.420377139 +0000 UTC m=+1501.892216986" watchObservedRunningTime="2026-09-17 16:03:56.421848974 +0000 UTC m=+1501.893688820"
Sep 17 16:03:56.826210 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:03:56.826116    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:04:01.826636 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:01.826582    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:04:01.839559 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:01.839525    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:04:02.423928 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:02.423895    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:04:27.396673 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:27.396584    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:04:40.155235 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:40.155198    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:04:40.255718 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:40.255688    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:04:40.495596 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:40.495481    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:04:43.711973 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:43.711908    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:04:43.798796 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:43.798757    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:04:44.683950 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:44.683920    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh_6557a20d-459b-4587-88e7-1d20bd3a0243/main/0.log"
Sep 17 16:04:45.616093 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:45.616060    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:04:45.631634 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:45.631601    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"]
Sep 17 16:04:45.631932 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:45.631888    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh" podUID="6557a20d-459b-4587-88e7-1d20bd3a0243" containerName="main" containerID="cri-o://c62e12557ed08cb28c9faeb35449f0892e5ab4e92ca732b2c4e9a3df9fcbe2f4" gracePeriod=30
Sep 17 16:04:45.639232 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:45.639204    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"]
Sep 17 16:04:45.648586 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:04:45.648535    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:04:45.648586 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:04:45.648586 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:04:45.648586 ip-10-0-129-37 kubenswrapper[2574]:  > containerID="c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7" cmd=["/bin/sleep","15"]
Sep 17 16:04:45.648883 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:45.648595    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj" message="" err=<
Sep 17 16:04:45.648883 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:04:45.648883 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:04:45.648883 ip-10-0-129-37 kubenswrapper[2574]:  >
Sep 17 16:04:45.648883 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:04:45.648608    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:04:45.648883 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:04:45.648883 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:04:45.648883 ip-10-0-129-37 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj" podUID="95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" containerName="main" containerID="cri-o://c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7"
Sep 17 16:04:45.648883 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:45.648672    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj" podUID="95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" containerName="main" containerID="cri-o://c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7" gracePeriod=60
Sep 17 16:04:45.691766 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:45.691729    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:04:46.003845 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.003820    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:04:46.113262 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.113140    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w7lt\" (UniqueName: \"kubernetes.io/projected/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-kube-api-access-6w7lt\") pod \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") "
Sep 17 16:04:46.113262 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.113207    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-model-cache\") pod \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") "
Sep 17 16:04:46.113262 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.113265    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-tls-certs\") pod \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") "
Sep 17 16:04:46.113582 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.113313    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-dshm\") pod \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") "
Sep 17 16:04:46.113582 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.113344    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-tmp-dir\") pod \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") "
Sep 17 16:04:46.113582 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.113383    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-kserve-provision-location\") pod \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") "
Sep 17 16:04:46.113582 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.113416    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-home\") pod \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") "
Sep 17 16:04:46.113582 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.113484    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-cabundle-cert\") pod \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\" (UID: \"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5\") "
Sep 17 16:04:46.113856 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.113796    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-home" (OuterVolumeSpecName: "home") pod "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" (UID: "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:04:46.113912 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.113867    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" (UID: "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:04:46.113970 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.113919    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-model-cache" (OuterVolumeSpecName: "model-cache") pod "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" (UID: "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:04:46.114203 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.114163    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" (UID: "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:04:46.116052 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.116021    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" (UID: "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:04:46.116052 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.116037    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-kube-api-access-6w7lt" (OuterVolumeSpecName: "kube-api-access-6w7lt") pod "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" (UID: "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5"). InnerVolumeSpecName "kube-api-access-6w7lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:04:46.116386 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.116361    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-dshm" (OuterVolumeSpecName: "dshm") pod "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" (UID: "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:04:46.137408 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.137370    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" (UID: "95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:04:46.203709 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.203683    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:04:46.214415 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.214385    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:04:46.214415 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.214407    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:04:46.214568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.214420    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:04:46.214568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.214433    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:04:46.214568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.214456    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:04:46.214568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.214465    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:04:46.214568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.214474    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w7lt\" (UniqueName: \"kubernetes.io/projected/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-kube-api-access-6w7lt\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:04:46.214568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.214482    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:04:46.314764 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.314729    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6557a20d-459b-4587-88e7-1d20bd3a0243-kserve-provision-location\") pod \"6557a20d-459b-4587-88e7-1d20bd3a0243\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") "
Sep 17 16:04:46.314951 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.314786    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6557a20d-459b-4587-88e7-1d20bd3a0243-cabundle-cert\") pod \"6557a20d-459b-4587-88e7-1d20bd3a0243\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") "
Sep 17 16:04:46.314951 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.314810    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6557a20d-459b-4587-88e7-1d20bd3a0243-tls-certs\") pod \"6557a20d-459b-4587-88e7-1d20bd3a0243\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") "
Sep 17 16:04:46.314951 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.314893    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6557a20d-459b-4587-88e7-1d20bd3a0243-tmp-dir\") pod \"6557a20d-459b-4587-88e7-1d20bd3a0243\" (UID: \"6557a20d-459b-4587-88e7-1d20bd3a0243\") "
Sep 17 16:04:46.315257 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.315165    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6557a20d-459b-4587-88e7-1d20bd3a0243-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6557a20d-459b-4587-88e7-1d20bd3a0243" (UID: "6557a20d-459b-4587-88e7-1d20bd3a0243"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:04:46.315431 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.315395    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6557a20d-459b-4587-88e7-1d20bd3a0243-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6557a20d-459b-4587-88e7-1d20bd3a0243" (UID: "6557a20d-459b-4587-88e7-1d20bd3a0243"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:04:46.317032 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.317007    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6557a20d-459b-4587-88e7-1d20bd3a0243-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6557a20d-459b-4587-88e7-1d20bd3a0243" (UID: "6557a20d-459b-4587-88e7-1d20bd3a0243"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:04:46.340635 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.340603    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6557a20d-459b-4587-88e7-1d20bd3a0243-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6557a20d-459b-4587-88e7-1d20bd3a0243" (UID: "6557a20d-459b-4587-88e7-1d20bd3a0243"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:04:46.416228 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.416192    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6557a20d-459b-4587-88e7-1d20bd3a0243-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:04:46.416228 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.416226    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6557a20d-459b-4587-88e7-1d20bd3a0243-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:04:46.416418 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.416241    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6557a20d-459b-4587-88e7-1d20bd3a0243-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:04:46.416418 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.416254    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6557a20d-459b-4587-88e7-1d20bd3a0243-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:04:46.628187 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.628153    2574 generic.go:356] "Generic (PLEG): container finished" podID="6557a20d-459b-4587-88e7-1d20bd3a0243" containerID="c62e12557ed08cb28c9faeb35449f0892e5ab4e92ca732b2c4e9a3df9fcbe2f4" exitCode=0
Sep 17 16:04:46.628620 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.628200    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"
Sep 17 16:04:46.628620 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.628240    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh" event={"ID":"6557a20d-459b-4587-88e7-1d20bd3a0243","Type":"ContainerDied","Data":"c62e12557ed08cb28c9faeb35449f0892e5ab4e92ca732b2c4e9a3df9fcbe2f4"}
Sep 17 16:04:46.628620 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.628289    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh" event={"ID":"6557a20d-459b-4587-88e7-1d20bd3a0243","Type":"ContainerDied","Data":"72f5cadd1e9c869e993853990d2156c1a0bc03cb013d833b0206f3527048aab8"}
Sep 17 16:04:46.628620 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.628314    2574 scope.go:117] "RemoveContainer" containerID="c62e12557ed08cb28c9faeb35449f0892e5ab4e92ca732b2c4e9a3df9fcbe2f4"
Sep 17 16:04:46.629932 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.629906    2574 generic.go:356] "Generic (PLEG): container finished" podID="95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" containerID="c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7" exitCode=0
Sep 17 16:04:46.630052 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.629951    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"
Sep 17 16:04:46.630052 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.629985    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj" event={"ID":"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5","Type":"ContainerDied","Data":"c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7"}
Sep 17 16:04:46.630052 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.630022    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj" event={"ID":"95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5","Type":"ContainerDied","Data":"402c6706bb444d22a01569e11fd3e983559eb75bef9571e8cb35ef347af93767"}
Sep 17 16:04:46.637386 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.637240    2574 scope.go:117] "RemoveContainer" containerID="33aa1c7de3a6433a0fe84d068e8e4ac71b002fd66056d153205e000f26ea35d7"
Sep 17 16:04:46.645290 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.645271    2574 scope.go:117] "RemoveContainer" containerID="c62e12557ed08cb28c9faeb35449f0892e5ab4e92ca732b2c4e9a3df9fcbe2f4"
Sep 17 16:04:46.645687 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:04:46.645660    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c62e12557ed08cb28c9faeb35449f0892e5ab4e92ca732b2c4e9a3df9fcbe2f4\": container with ID starting with c62e12557ed08cb28c9faeb35449f0892e5ab4e92ca732b2c4e9a3df9fcbe2f4 not found: ID does not exist" containerID="c62e12557ed08cb28c9faeb35449f0892e5ab4e92ca732b2c4e9a3df9fcbe2f4"
Sep 17 16:04:46.645760 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.645697    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62e12557ed08cb28c9faeb35449f0892e5ab4e92ca732b2c4e9a3df9fcbe2f4"} err="failed to get container status \"c62e12557ed08cb28c9faeb35449f0892e5ab4e92ca732b2c4e9a3df9fcbe2f4\": rpc error: code = NotFound desc = could not find container \"c62e12557ed08cb28c9faeb35449f0892e5ab4e92ca732b2c4e9a3df9fcbe2f4\": container with ID starting with c62e12557ed08cb28c9faeb35449f0892e5ab4e92ca732b2c4e9a3df9fcbe2f4 not found: ID does not exist"
Sep 17 16:04:46.645760 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.645719    2574 scope.go:117] "RemoveContainer" containerID="33aa1c7de3a6433a0fe84d068e8e4ac71b002fd66056d153205e000f26ea35d7"
Sep 17 16:04:46.646009 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:04:46.645993    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33aa1c7de3a6433a0fe84d068e8e4ac71b002fd66056d153205e000f26ea35d7\": container with ID starting with 33aa1c7de3a6433a0fe84d068e8e4ac71b002fd66056d153205e000f26ea35d7 not found: ID does not exist" containerID="33aa1c7de3a6433a0fe84d068e8e4ac71b002fd66056d153205e000f26ea35d7"
Sep 17 16:04:46.646055 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.646016    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33aa1c7de3a6433a0fe84d068e8e4ac71b002fd66056d153205e000f26ea35d7"} err="failed to get container status \"33aa1c7de3a6433a0fe84d068e8e4ac71b002fd66056d153205e000f26ea35d7\": rpc error: code = NotFound desc = could not find container \"33aa1c7de3a6433a0fe84d068e8e4ac71b002fd66056d153205e000f26ea35d7\": container with ID starting with 33aa1c7de3a6433a0fe84d068e8e4ac71b002fd66056d153205e000f26ea35d7 not found: ID does not exist"
Sep 17 16:04:46.646055 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.646031    2574 scope.go:117] "RemoveContainer" containerID="c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7"
Sep 17 16:04:46.653794 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.653775    2574 scope.go:117] "RemoveContainer" containerID="4a3c5249369bdc806b3e330685db35785b8d351773e29719f23def8cdd6dab41"
Sep 17 16:04:46.654689 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.654677    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"]
Sep 17 16:04:46.658931 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.658906    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-58d9564556-xc6xh"]
Sep 17 16:04:46.661666 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.661650    2574 scope.go:117] "RemoveContainer" containerID="c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7"
Sep 17 16:04:46.661916 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:04:46.661899    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7\": container with ID starting with c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7 not found: ID does not exist" containerID="c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7"
Sep 17 16:04:46.661987 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.661925    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7"} err="failed to get container status \"c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7\": rpc error: code = NotFound desc = could not find container \"c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7\": container with ID starting with c97d4262cfc2d42fb03181d641a2f22e1a42ff9c4e6d1fffccb4af726dea97c7 not found: ID does not exist"
Sep 17 16:04:46.661987 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.661946    2574 scope.go:117] "RemoveContainer" containerID="4a3c5249369bdc806b3e330685db35785b8d351773e29719f23def8cdd6dab41"
Sep 17 16:04:46.662190 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:04:46.662172    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3c5249369bdc806b3e330685db35785b8d351773e29719f23def8cdd6dab41\": container with ID starting with 4a3c5249369bdc806b3e330685db35785b8d351773e29719f23def8cdd6dab41 not found: ID does not exist" containerID="4a3c5249369bdc806b3e330685db35785b8d351773e29719f23def8cdd6dab41"
Sep 17 16:04:46.662235 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.662196    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3c5249369bdc806b3e330685db35785b8d351773e29719f23def8cdd6dab41"} err="failed to get container status \"4a3c5249369bdc806b3e330685db35785b8d351773e29719f23def8cdd6dab41\": rpc error: code = NotFound desc = could not find container \"4a3c5249369bdc806b3e330685db35785b8d351773e29719f23def8cdd6dab41\": container with ID starting with 4a3c5249369bdc806b3e330685db35785b8d351773e29719f23def8cdd6dab41 not found: ID does not exist"
Sep 17 16:04:46.669054 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.669028    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"]
Sep 17 16:04:46.673327 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:46.673308    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5994fd5b86-nvrdj"]
Sep 17 16:04:47.049943 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:47.049860    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6557a20d-459b-4587-88e7-1d20bd3a0243" path="/var/lib/kubelet/pods/6557a20d-459b-4587-88e7-1d20bd3a0243/volumes"
Sep 17 16:04:47.050331 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:04:47.050313    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" path="/var/lib/kubelet/pods/95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5/volumes"
Sep 17 16:05:33.612176 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:05:33.612132    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:05:33.709717 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:05:33.709676    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:05:35.007014 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:05:35.006966    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:05:35.103808 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:05:35.103768    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:05:40.431568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:05:40.431533    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:05:41.686511 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:05:41.686473    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:05:46.536715 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:05:46.536665    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:05:48.591920 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:05:48.591840    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:10.679700 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.679659    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"]
Sep 17 16:06:10.680082 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.680055    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" containerName="storage-initializer"
Sep 17 16:06:10.680082 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.680067    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" containerName="storage-initializer"
Sep 17 16:06:10.680082 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.680080    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6557a20d-459b-4587-88e7-1d20bd3a0243" containerName="storage-initializer"
Sep 17 16:06:10.680197 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.680086    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6557a20d-459b-4587-88e7-1d20bd3a0243" containerName="storage-initializer"
Sep 17 16:06:10.680197 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.680094    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6557a20d-459b-4587-88e7-1d20bd3a0243" containerName="main"
Sep 17 16:06:10.680197 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.680103    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6557a20d-459b-4587-88e7-1d20bd3a0243" containerName="main"
Sep 17 16:06:10.680197 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.680115    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" containerName="main"
Sep 17 16:06:10.680197 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.680121    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" containerName="main"
Sep 17 16:06:10.680197 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.680174    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6557a20d-459b-4587-88e7-1d20bd3a0243" containerName="main"
Sep 17 16:06:10.680197 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.680181    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="95991c50-b4f9-40c8-8a12-9a7b4a6b4ac5" containerName="main"
Sep 17 16:06:10.683423 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.683404    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.686725 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.686697    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 16:06:10.686887 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.686783    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 16:06:10.687322 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.686929    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-bkxzz\""
Sep 17 16:06:10.687322 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.686956    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 16:06:10.687322 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.687025    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 17 16:06:10.687561 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.687426    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:06:10.699729 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.699697    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"]
Sep 17 16:06:10.706761 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.706727    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:10.756135 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.756090    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtrpv\" (UniqueName: \"kubernetes.io/projected/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-kube-api-access-wtrpv\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.756135 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.756132    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-kserve-provision-location\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.756351 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.756224    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-model-cache\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.756351 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.756276    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-home\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.756351 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.756297    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-tmp-dir\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.756351 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.756330    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-cabundle-cert\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.756351 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.756347    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-dshm\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.756537 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.756368    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-tls-certs\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.857821 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.857772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-kserve-provision-location\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.858009 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.857844    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-model-cache\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.858009 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.857882    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-home\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.858009 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.857903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-tmp-dir\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.858009 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.857935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-cabundle-cert\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.858009 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.857952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-dshm\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.858009 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.857979    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-tls-certs\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.858337 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.858013    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtrpv\" (UniqueName: \"kubernetes.io/projected/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-kube-api-access-wtrpv\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.858337 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.858236    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-kserve-provision-location\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.858337 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.858276    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-home\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.858337 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.858304    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-model-cache\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.858598 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.858331    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-tmp-dir\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.858703 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.858681    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-cabundle-cert\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.860512 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.860489    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-dshm\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.860629 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.860594    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-tls-certs\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.870379 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.870354    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtrpv\" (UniqueName: \"kubernetes.io/projected/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-kube-api-access-wtrpv\") pod \"auth-disabled-test-kserve-6d9dcb8b44-vzwjm\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:10.965069 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.964973    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:10.994246 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:10.994200    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:11.278568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:11.278542    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"]
Sep 17 16:06:11.280067 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:06:11.280041    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270dce6e_6415_45aa_99dd_e9ded1e8a0ac.slice/crio-3485ef3bf2acdf4c819bd155a5dda8ab65a3812709947f15499d3a3a58b80fb6 WatchSource:0}: Error finding container 3485ef3bf2acdf4c819bd155a5dda8ab65a3812709947f15499d3a3a58b80fb6: Status 404 returned error can't find the container with id 3485ef3bf2acdf4c819bd155a5dda8ab65a3812709947f15499d3a3a58b80fb6
Sep 17 16:06:12.053738 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:12.053691    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" event={"ID":"270dce6e-6415-45aa-99dd-e9ded1e8a0ac","Type":"ContainerStarted","Data":"781636230c8b04fad5f63bcd681471e1f05368f5b90d8253fb2fe5838a63420d"}
Sep 17 16:06:12.054151 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:12.053780    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" event={"ID":"270dce6e-6415-45aa-99dd-e9ded1e8a0ac","Type":"ContainerStarted","Data":"3485ef3bf2acdf4c819bd155a5dda8ab65a3812709947f15499d3a3a58b80fb6"}
Sep 17 16:06:14.064580 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:14.064496    2574 generic.go:356] "Generic (PLEG): container finished" podID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerID="781636230c8b04fad5f63bcd681471e1f05368f5b90d8253fb2fe5838a63420d" exitCode=0
Sep 17 16:06:14.064580 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:14.064552    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" event={"ID":"270dce6e-6415-45aa-99dd-e9ded1e8a0ac","Type":"ContainerDied","Data":"781636230c8b04fad5f63bcd681471e1f05368f5b90d8253fb2fe5838a63420d"}
Sep 17 16:06:15.072010 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:15.071969    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" event={"ID":"270dce6e-6415-45aa-99dd-e9ded1e8a0ac","Type":"ContainerStarted","Data":"919b516d201af99df43e5a5393c6b4593d12a6f0747bb315bfa918c550fc1de1"}
Sep 17 16:06:15.100060 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:15.099995    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" podStartSLOduration=2.704180923 podStartE2EDuration="5.099977283s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.39579636s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:06:15.098540274 +0000 UTC m=+1640.570380125" watchObservedRunningTime="2026-09-17 16:06:15.099977283 +0000 UTC m=+1640.571817129"
Sep 17 16:06:15.994363 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:15.994326    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:18.522351 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:18.522311    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:20.995272 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:20.995229    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:06:20.996788 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:20.996755    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 16:06:24.310894 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.310857    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"]
Sep 17 16:06:24.313774 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.313754    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.316311 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.316278    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 16:06:24.316768 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.316746    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-fcrqx\""
Sep 17 16:06:24.316987 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.316779    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 16:06:24.316987 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.316745    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 16:06:24.317150 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.317131    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 16:06:24.317210 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.317171    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:06:24.328899 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.328869    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"]
Sep 17 16:06:24.329089 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.328870    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:24.386195 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.386162    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.386195 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.386201    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.386452 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.386231    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-model-cache\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.386452 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.386267    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.386452 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.386351    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-dshm\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.386452 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.386383    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-home\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.386452 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.386402    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-tls-certs\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.386628 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.386455    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmzrs\" (UniqueName: \"kubernetes.io/projected/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-kube-api-access-gmzrs\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.487528 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.487476    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-model-cache\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.487528 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.487532    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.487772 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.487568    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-dshm\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.487772 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.487596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-home\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.487772 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.487618    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-tls-certs\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.487772 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.487641    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmzrs\" (UniqueName: \"kubernetes.io/projected/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-kube-api-access-gmzrs\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.487982 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.487785    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.487982 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.487810    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.487982 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.487938    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-model-cache\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.487982 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.487960    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.488201 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.488043    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-home\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.488259 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.488220    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.488512 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.488490    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.490373 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.490352    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-tls-certs\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.490601 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.490583    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-dshm\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.497066 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.497041    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmzrs\" (UniqueName: \"kubernetes.io/projected/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-kube-api-access-gmzrs\") pod \"tokenizer-template-args-test-kserve-6dd775764b-mlxgn\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.534954 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.534913    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"]
Sep 17 16:06:24.539268 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.539239    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:24.549923 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.549784    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"]
Sep 17 16:06:24.566480 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.566363    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:24.625824 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.625777    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:24.645739 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.645704    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:24.689771 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.689731    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b9c8d9-1887-4009-8872-1de1d16503e1-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-777b587468-s5x7m\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:24.689929 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.689811    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b9c8d9-1887-4009-8872-1de1d16503e1-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-777b587468-s5x7m\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:24.689929 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.689866    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b9c8d9-1887-4009-8872-1de1d16503e1-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-777b587468-s5x7m\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:24.690024 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.689926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b9c8d9-1887-4009-8872-1de1d16503e1-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-777b587468-s5x7m\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:24.790959 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.790908    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b9c8d9-1887-4009-8872-1de1d16503e1-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-777b587468-s5x7m\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:24.791181 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.790986    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b9c8d9-1887-4009-8872-1de1d16503e1-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-777b587468-s5x7m\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:24.791181 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.791037    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b9c8d9-1887-4009-8872-1de1d16503e1-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-777b587468-s5x7m\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:24.791181 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.791087    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b9c8d9-1887-4009-8872-1de1d16503e1-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-777b587468-s5x7m\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:24.791416 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.791385    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b9c8d9-1887-4009-8872-1de1d16503e1-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-777b587468-s5x7m\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:24.791576 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.791459    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b9c8d9-1887-4009-8872-1de1d16503e1-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-777b587468-s5x7m\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:24.791870 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.791851    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b9c8d9-1887-4009-8872-1de1d16503e1-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-777b587468-s5x7m\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:24.793658 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.793637    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b9c8d9-1887-4009-8872-1de1d16503e1-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-777b587468-s5x7m\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:24.851928 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.851891    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:24.915494 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:24.912324    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"]
Sep 17 16:06:24.919165 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:06:24.919042    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f875e3_32fa_4b33_aa42_fb26d96c71d3.slice/crio-33c49895df5475fa3d498ce43160ffbf0ae89e5126fa14dad758460c11b153df WatchSource:0}: Error finding container 33c49895df5475fa3d498ce43160ffbf0ae89e5126fa14dad758460c11b153df: Status 404 returned error can't find the container with id 33c49895df5475fa3d498ce43160ffbf0ae89e5126fa14dad758460c11b153df
Sep 17 16:06:25.126698 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:25.126589    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn" event={"ID":"e3f875e3-32fa-4b33-aa42-fb26d96c71d3","Type":"ContainerStarted","Data":"4fb672a222aaf0aefffe5b0bb70bf72548a79187ee3b2aff3fcec76234507edf"}
Sep 17 16:06:25.126698 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:25.126658    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn" event={"ID":"e3f875e3-32fa-4b33-aa42-fb26d96c71d3","Type":"ContainerStarted","Data":"33c49895df5475fa3d498ce43160ffbf0ae89e5126fa14dad758460c11b153df"}
Sep 17 16:06:25.134662 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:25.134629    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"]
Sep 17 16:06:25.138632 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:06:25.138595    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b9c8d9_1887_4009_8872_1de1d16503e1.slice/crio-d7cce246379387f39d57125c727842ab5231ceca626b869951bed2570cd14af5 WatchSource:0}: Error finding container d7cce246379387f39d57125c727842ab5231ceca626b869951bed2570cd14af5: Status 404 returned error can't find the container with id d7cce246379387f39d57125c727842ab5231ceca626b869951bed2570cd14af5
Sep 17 16:06:26.137678 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:26.137631    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m" event={"ID":"51b9c8d9-1887-4009-8872-1de1d16503e1","Type":"ContainerStarted","Data":"ce1a0fc7bcba08553821913926f6d555551ce6b1baa3d56f9ee770417671e554"}
Sep 17 16:06:26.138128 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:26.137703    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m" event={"ID":"51b9c8d9-1887-4009-8872-1de1d16503e1","Type":"ContainerStarted","Data":"d7cce246379387f39d57125c727842ab5231ceca626b869951bed2570cd14af5"}
Sep 17 16:06:28.154021 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:28.153990    2574 generic.go:356] "Generic (PLEG): container finished" podID="e3f875e3-32fa-4b33-aa42-fb26d96c71d3" containerID="4fb672a222aaf0aefffe5b0bb70bf72548a79187ee3b2aff3fcec76234507edf" exitCode=0
Sep 17 16:06:28.154475 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:28.154060    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn" event={"ID":"e3f875e3-32fa-4b33-aa42-fb26d96c71d3","Type":"ContainerDied","Data":"4fb672a222aaf0aefffe5b0bb70bf72548a79187ee3b2aff3fcec76234507edf"}
Sep 17 16:06:28.155759 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:28.155734    2574 generic.go:356] "Generic (PLEG): container finished" podID="51b9c8d9-1887-4009-8872-1de1d16503e1" containerID="ce1a0fc7bcba08553821913926f6d555551ce6b1baa3d56f9ee770417671e554" exitCode=0
Sep 17 16:06:28.155864 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:28.155791    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m" event={"ID":"51b9c8d9-1887-4009-8872-1de1d16503e1","Type":"ContainerDied","Data":"ce1a0fc7bcba08553821913926f6d555551ce6b1baa3d56f9ee770417671e554"}
Sep 17 16:06:29.164039 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:29.163993    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m" event={"ID":"51b9c8d9-1887-4009-8872-1de1d16503e1","Type":"ContainerStarted","Data":"c654ecafbacdd5b7287f9397c95e2a5d8065b7081803c91d980176d15498af59"}
Sep 17 16:06:29.164541 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:29.164233    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:06:29.166246 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:29.166217    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn" event={"ID":"e3f875e3-32fa-4b33-aa42-fb26d96c71d3","Type":"ContainerStarted","Data":"73451b56204ea743e294a2796955f0eedd001fe11e6c8398f078032aae1488ab"}
Sep 17 16:06:29.192452 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:29.192396    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m" podStartSLOduration=2.419103774 podStartE2EDuration="5.192378554s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.77327478s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:06:29.191044823 +0000 UTC m=+1654.662884669" watchObservedRunningTime="2026-09-17 16:06:29.192378554 +0000 UTC m=+1654.664218400"
Sep 17 16:06:29.236057 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:29.235995    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn" podStartSLOduration=2.445251449 podStartE2EDuration="5.23597316s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.790721711s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:06:29.23295658 +0000 UTC m=+1654.704796449" watchObservedRunningTime="2026-09-17 16:06:29.23597316 +0000 UTC m=+1654.707813007"
Sep 17 16:06:29.626022 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:29.625989    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:30.995155 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:30.995102    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 16:06:34.626673 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:34.626628    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:34.639604 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:34.639570    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:35.208110 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:35.208079    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:06:36.872493 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:36.872454    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:40.995636 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:40.995595    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 16:06:50.994919 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:06:50.994867    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 16:07:00.182427 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:00.182392    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:07:00.994723 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:00.994674    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 16:07:01.441836 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:01.441801    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-777b587468-s5x7m_51b9c8d9-1887-4009-8872-1de1d16503e1/main/0.log"
Sep 17 16:07:02.037629 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.037583    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:07:02.100843 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.100800    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"]
Sep 17 16:07:02.111810 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:07:02.111766    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:07:02.111810 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:07:02.111810 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:07:02.111810 ip-10-0-129-37 kubenswrapper[2574]:  > containerID="73451b56204ea743e294a2796955f0eedd001fe11e6c8398f078032aae1488ab" cmd=["/bin/sleep","15"]
Sep 17 16:07:02.112169 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.111839    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn" message="" err=<
Sep 17 16:07:02.112169 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:07:02.112169 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:07:02.112169 ip-10-0-129-37 kubenswrapper[2574]:  >
Sep 17 16:07:02.112169 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:07:02.111856    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:07:02.112169 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:07:02.112169 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:07:02.112169 ip-10-0-129-37 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn" podUID="e3f875e3-32fa-4b33-aa42-fb26d96c71d3" containerName="main" containerID="cri-o://73451b56204ea743e294a2796955f0eedd001fe11e6c8398f078032aae1488ab"
Sep 17 16:07:02.112169 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.111906    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn" podUID="e3f875e3-32fa-4b33-aa42-fb26d96c71d3" containerName="main" containerID="cri-o://73451b56204ea743e294a2796955f0eedd001fe11e6c8398f078032aae1488ab" gracePeriod=60
Sep 17 16:07:02.123726 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.123630    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"]
Sep 17 16:07:02.123924 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.123897    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m" podUID="51b9c8d9-1887-4009-8872-1de1d16503e1" containerName="main" containerID="cri-o://c654ecafbacdd5b7287f9397c95e2a5d8065b7081803c91d980176d15498af59" gracePeriod=30
Sep 17 16:07:02.196797 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.196745    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn" podUID="e3f875e3-32fa-4b33-aa42-fb26d96c71d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 17 16:07:02.238576 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.238545    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:07:02.331011 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.330977    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:07:02.345205 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.345165    2574 generic.go:356] "Generic (PLEG): container finished" podID="e3f875e3-32fa-4b33-aa42-fb26d96c71d3" containerID="73451b56204ea743e294a2796955f0eedd001fe11e6c8398f078032aae1488ab" exitCode=0
Sep 17 16:07:02.345380 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.345353    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn" event={"ID":"e3f875e3-32fa-4b33-aa42-fb26d96c71d3","Type":"ContainerDied","Data":"73451b56204ea743e294a2796955f0eedd001fe11e6c8398f078032aae1488ab"}
Sep 17 16:07:02.480401 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.480370    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:07:02.576775 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.576698    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-tmp-dir\") pod \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") "
Sep 17 16:07:02.576775 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.576749    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-kserve-provision-location\") pod \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") "
Sep 17 16:07:02.576965 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.576788    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-model-cache\") pod \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") "
Sep 17 16:07:02.576965 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.576864    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-cabundle-cert\") pod \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") "
Sep 17 16:07:02.576965 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.576888    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmzrs\" (UniqueName: \"kubernetes.io/projected/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-kube-api-access-gmzrs\") pod \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") "
Sep 17 16:07:02.576965 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.576915    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-tls-certs\") pod \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") "
Sep 17 16:07:02.576965 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.576942    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-dshm\") pod \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") "
Sep 17 16:07:02.577211 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.577024    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-home\") pod \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\" (UID: \"e3f875e3-32fa-4b33-aa42-fb26d96c71d3\") "
Sep 17 16:07:02.577211 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.577028    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3f875e3-32fa-4b33-aa42-fb26d96c71d3" (UID: "e3f875e3-32fa-4b33-aa42-fb26d96c71d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:07:02.577211 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.577076    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3f875e3-32fa-4b33-aa42-fb26d96c71d3" (UID: "e3f875e3-32fa-4b33-aa42-fb26d96c71d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:07:02.577374 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.577299    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:07:02.577374 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.577325    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:07:02.577475 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.577393    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e3f875e3-32fa-4b33-aa42-fb26d96c71d3" (UID: "e3f875e3-32fa-4b33-aa42-fb26d96c71d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:07:02.577548 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.577525    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-home" (OuterVolumeSpecName: "home") pod "e3f875e3-32fa-4b33-aa42-fb26d96c71d3" (UID: "e3f875e3-32fa-4b33-aa42-fb26d96c71d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:07:02.594861 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.594822    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-kube-api-access-gmzrs" (OuterVolumeSpecName: "kube-api-access-gmzrs") pod "e3f875e3-32fa-4b33-aa42-fb26d96c71d3" (UID: "e3f875e3-32fa-4b33-aa42-fb26d96c71d3"). InnerVolumeSpecName "kube-api-access-gmzrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:07:02.609221 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.609182    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3f875e3-32fa-4b33-aa42-fb26d96c71d3" (UID: "e3f875e3-32fa-4b33-aa42-fb26d96c71d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:07:02.609878 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.609850    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-dshm" (OuterVolumeSpecName: "dshm") pod "e3f875e3-32fa-4b33-aa42-fb26d96c71d3" (UID: "e3f875e3-32fa-4b33-aa42-fb26d96c71d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:07:02.609972 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.609866    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3f875e3-32fa-4b33-aa42-fb26d96c71d3" (UID: "e3f875e3-32fa-4b33-aa42-fb26d96c71d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:07:02.678802 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.678765    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:07:02.678802 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.678800    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:07:02.679008 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.678811    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmzrs\" (UniqueName: \"kubernetes.io/projected/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-kube-api-access-gmzrs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:07:02.679008 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.678821    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:07:02.679008 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.678831    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:07:02.679008 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.678838    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f875e3-32fa-4b33-aa42-fb26d96c71d3-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:07:02.777414 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.777383    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:07:02.880266 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.880224    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b9c8d9-1887-4009-8872-1de1d16503e1-tmp-dir\") pod \"51b9c8d9-1887-4009-8872-1de1d16503e1\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") "
Sep 17 16:07:02.880266 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.880274    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b9c8d9-1887-4009-8872-1de1d16503e1-kserve-provision-location\") pod \"51b9c8d9-1887-4009-8872-1de1d16503e1\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") "
Sep 17 16:07:02.880578 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.880312    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b9c8d9-1887-4009-8872-1de1d16503e1-tls-certs\") pod \"51b9c8d9-1887-4009-8872-1de1d16503e1\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") "
Sep 17 16:07:02.880578 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.880332    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b9c8d9-1887-4009-8872-1de1d16503e1-cabundle-cert\") pod \"51b9c8d9-1887-4009-8872-1de1d16503e1\" (UID: \"51b9c8d9-1887-4009-8872-1de1d16503e1\") "
Sep 17 16:07:02.880723 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.880693    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b9c8d9-1887-4009-8872-1de1d16503e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51b9c8d9-1887-4009-8872-1de1d16503e1" (UID: "51b9c8d9-1887-4009-8872-1de1d16503e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:07:02.880788 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.880732    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b9c8d9-1887-4009-8872-1de1d16503e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51b9c8d9-1887-4009-8872-1de1d16503e1" (UID: "51b9c8d9-1887-4009-8872-1de1d16503e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:07:02.882460 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.882421    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b9c8d9-1887-4009-8872-1de1d16503e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51b9c8d9-1887-4009-8872-1de1d16503e1" (UID: "51b9c8d9-1887-4009-8872-1de1d16503e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:07:02.900871 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.900823    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b9c8d9-1887-4009-8872-1de1d16503e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51b9c8d9-1887-4009-8872-1de1d16503e1" (UID: "51b9c8d9-1887-4009-8872-1de1d16503e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:07:02.981750 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.981708    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b9c8d9-1887-4009-8872-1de1d16503e1-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:07:02.981750 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.981737    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b9c8d9-1887-4009-8872-1de1d16503e1-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:07:02.981750 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.981748    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b9c8d9-1887-4009-8872-1de1d16503e1-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:07:02.981750 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:02.981757    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b9c8d9-1887-4009-8872-1de1d16503e1-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:07:03.351340 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.351311    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"
Sep 17 16:07:03.351517 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.351319    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn" event={"ID":"e3f875e3-32fa-4b33-aa42-fb26d96c71d3","Type":"ContainerDied","Data":"33c49895df5475fa3d498ce43160ffbf0ae89e5126fa14dad758460c11b153df"}
Sep 17 16:07:03.351517 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.351490    2574 scope.go:117] "RemoveContainer" containerID="73451b56204ea743e294a2796955f0eedd001fe11e6c8398f078032aae1488ab"
Sep 17 16:07:03.353522 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.353490    2574 generic.go:356] "Generic (PLEG): container finished" podID="51b9c8d9-1887-4009-8872-1de1d16503e1" containerID="c654ecafbacdd5b7287f9397c95e2a5d8065b7081803c91d980176d15498af59" exitCode=0
Sep 17 16:07:03.353664 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.353598    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"
Sep 17 16:07:03.353664 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.353642    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m" event={"ID":"51b9c8d9-1887-4009-8872-1de1d16503e1","Type":"ContainerDied","Data":"c654ecafbacdd5b7287f9397c95e2a5d8065b7081803c91d980176d15498af59"}
Sep 17 16:07:03.353778 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.353684    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m" event={"ID":"51b9c8d9-1887-4009-8872-1de1d16503e1","Type":"ContainerDied","Data":"d7cce246379387f39d57125c727842ab5231ceca626b869951bed2570cd14af5"}
Sep 17 16:07:03.360783 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.360763    2574 scope.go:117] "RemoveContainer" containerID="4fb672a222aaf0aefffe5b0bb70bf72548a79187ee3b2aff3fcec76234507edf"
Sep 17 16:07:03.370768 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.370743    2574 scope.go:117] "RemoveContainer" containerID="c654ecafbacdd5b7287f9397c95e2a5d8065b7081803c91d980176d15498af59"
Sep 17 16:07:03.378594 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.378571    2574 scope.go:117] "RemoveContainer" containerID="ce1a0fc7bcba08553821913926f6d555551ce6b1baa3d56f9ee770417671e554"
Sep 17 16:07:03.384011 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.383978    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"]
Sep 17 16:07:03.387537 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.387508    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-777b587468-s5x7m"]
Sep 17 16:07:03.388285 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.388265    2574 scope.go:117] "RemoveContainer" containerID="c654ecafbacdd5b7287f9397c95e2a5d8065b7081803c91d980176d15498af59"
Sep 17 16:07:03.388622 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:07:03.388592    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c654ecafbacdd5b7287f9397c95e2a5d8065b7081803c91d980176d15498af59\": container with ID starting with c654ecafbacdd5b7287f9397c95e2a5d8065b7081803c91d980176d15498af59 not found: ID does not exist" containerID="c654ecafbacdd5b7287f9397c95e2a5d8065b7081803c91d980176d15498af59"
Sep 17 16:07:03.388703 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.388630    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c654ecafbacdd5b7287f9397c95e2a5d8065b7081803c91d980176d15498af59"} err="failed to get container status \"c654ecafbacdd5b7287f9397c95e2a5d8065b7081803c91d980176d15498af59\": rpc error: code = NotFound desc = could not find container \"c654ecafbacdd5b7287f9397c95e2a5d8065b7081803c91d980176d15498af59\": container with ID starting with c654ecafbacdd5b7287f9397c95e2a5d8065b7081803c91d980176d15498af59 not found: ID does not exist"
Sep 17 16:07:03.388703 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.388649    2574 scope.go:117] "RemoveContainer" containerID="ce1a0fc7bcba08553821913926f6d555551ce6b1baa3d56f9ee770417671e554"
Sep 17 16:07:03.388982 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:07:03.388952    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce1a0fc7bcba08553821913926f6d555551ce6b1baa3d56f9ee770417671e554\": container with ID starting with ce1a0fc7bcba08553821913926f6d555551ce6b1baa3d56f9ee770417671e554 not found: ID does not exist" containerID="ce1a0fc7bcba08553821913926f6d555551ce6b1baa3d56f9ee770417671e554"
Sep 17 16:07:03.389047 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.388991    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce1a0fc7bcba08553821913926f6d555551ce6b1baa3d56f9ee770417671e554"} err="failed to get container status \"ce1a0fc7bcba08553821913926f6d555551ce6b1baa3d56f9ee770417671e554\": rpc error: code = NotFound desc = could not find container \"ce1a0fc7bcba08553821913926f6d555551ce6b1baa3d56f9ee770417671e554\": container with ID starting with ce1a0fc7bcba08553821913926f6d555551ce6b1baa3d56f9ee770417671e554 not found: ID does not exist"
Sep 17 16:07:03.402189 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.402164    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"]
Sep 17 16:07:03.406538 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:03.406513    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6dd775764b-mlxgn"]
Sep 17 16:07:05.055501 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:05.055470    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51b9c8d9-1887-4009-8872-1de1d16503e1" path="/var/lib/kubelet/pods/51b9c8d9-1887-4009-8872-1de1d16503e1/volumes"
Sep 17 16:07:05.055953 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:05.055940    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f875e3-32fa-4b33-aa42-fb26d96c71d3" path="/var/lib/kubelet/pods/e3f875e3-32fa-4b33-aa42-fb26d96c71d3/volumes"
Sep 17 16:07:10.995354 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:10.995308    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 16:07:20.995356 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:20.995267    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 16:07:30.994948 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:30.994898    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 16:07:40.995063 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:40.995013    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 16:07:50.995044 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:50.994984    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 16:07:51.769906 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:51.769869    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:07:51.863162 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:51.863119    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:07:56.384188 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.384155    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"]
Sep 17 16:07:56.384598 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.384555    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f875e3-32fa-4b33-aa42-fb26d96c71d3" containerName="storage-initializer"
Sep 17 16:07:56.384598 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.384568    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f875e3-32fa-4b33-aa42-fb26d96c71d3" containerName="storage-initializer"
Sep 17 16:07:56.384598 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.384576    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b9c8d9-1887-4009-8872-1de1d16503e1" containerName="storage-initializer"
Sep 17 16:07:56.384598 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.384581    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b9c8d9-1887-4009-8872-1de1d16503e1" containerName="storage-initializer"
Sep 17 16:07:56.384598 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.384593    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f875e3-32fa-4b33-aa42-fb26d96c71d3" containerName="main"
Sep 17 16:07:56.384598 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.384599    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f875e3-32fa-4b33-aa42-fb26d96c71d3" containerName="main"
Sep 17 16:07:56.384812 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.384629    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b9c8d9-1887-4009-8872-1de1d16503e1" containerName="main"
Sep 17 16:07:56.384812 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.384635    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b9c8d9-1887-4009-8872-1de1d16503e1" containerName="main"
Sep 17 16:07:56.384812 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.384692    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f875e3-32fa-4b33-aa42-fb26d96c71d3" containerName="main"
Sep 17 16:07:56.384812 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.384700    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b9c8d9-1887-4009-8872-1de1d16503e1" containerName="main"
Sep 17 16:07:56.389938 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.389918    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.392935 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.392907    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 16:07:56.392935 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.392926    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 16:07:56.393181 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.393006    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-xjkk4\""
Sep 17 16:07:56.393256 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.393171    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 16:07:56.398093 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.397451    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"]
Sep 17 16:07:56.419161 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.419131    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:07:56.487218 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.487171    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.487405 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.487264    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsm4l\" (UniqueName: \"kubernetes.io/projected/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-kube-api-access-tsm4l\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.487405 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.487311    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.487405 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.487349    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.487628 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.487463    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.487628 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.487497    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.588644 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.588605    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsm4l\" (UniqueName: \"kubernetes.io/projected/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-kube-api-access-tsm4l\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.588935 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.588914    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.589086 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.589071    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.589210 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.589193    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.589327 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.589313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.589515 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.589496    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.589897 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.589870    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.590112 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.590093    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.590354 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.590321    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.592335 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.592289    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.592705 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.592683    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.605324 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.605289    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsm4l\" (UniqueName: \"kubernetes.io/projected/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-kube-api-access-tsm4l\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.609796 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.609767    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"]
Sep 17 16:07:56.616752 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.616728    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"
Sep 17 16:07:56.619732 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.619714    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-4n46k\""
Sep 17 16:07:56.625550 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.625523    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"]
Sep 17 16:07:56.635468 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.635349    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:07:56.702267 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.702225    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:56.791248 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.791186    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7582e2b0-c40b-4305-bd4b-23e66cb62f57-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv\" (UID: \"7582e2b0-c40b-4305-bd4b-23e66cb62f57\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"
Sep 17 16:07:56.791412 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.791256    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssdxr\" (UniqueName: \"kubernetes.io/projected/7582e2b0-c40b-4305-bd4b-23e66cb62f57-kube-api-access-ssdxr\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv\" (UID: \"7582e2b0-c40b-4305-bd4b-23e66cb62f57\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"
Sep 17 16:07:56.892263 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.892155    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7582e2b0-c40b-4305-bd4b-23e66cb62f57-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv\" (UID: \"7582e2b0-c40b-4305-bd4b-23e66cb62f57\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"
Sep 17 16:07:56.892263 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.892216    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssdxr\" (UniqueName: \"kubernetes.io/projected/7582e2b0-c40b-4305-bd4b-23e66cb62f57-kube-api-access-ssdxr\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv\" (UID: \"7582e2b0-c40b-4305-bd4b-23e66cb62f57\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"
Sep 17 16:07:56.894785 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.894763    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7582e2b0-c40b-4305-bd4b-23e66cb62f57-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv\" (UID: \"7582e2b0-c40b-4305-bd4b-23e66cb62f57\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"
Sep 17 16:07:56.901514 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.901492    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssdxr\" (UniqueName: \"kubernetes.io/projected/7582e2b0-c40b-4305-bd4b-23e66cb62f57-kube-api-access-ssdxr\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv\" (UID: \"7582e2b0-c40b-4305-bd4b-23e66cb62f57\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"
Sep 17 16:07:56.928889 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.928851    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"
Sep 17 16:07:56.982616 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:56.982586    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"]
Sep 17 16:07:56.993888 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:07:56.991867    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9189f04f_771a_4b3d_bff6_e1b8db10b4d5.slice/crio-34df4daf2729e84c931b345b057a39e1a940488f99885d01b8c80c4a6cef847b WatchSource:0}: Error finding container 34df4daf2729e84c931b345b057a39e1a940488f99885d01b8c80c4a6cef847b: Status 404 returned error can't find the container with id 34df4daf2729e84c931b345b057a39e1a940488f99885d01b8c80c4a6cef847b
Sep 17 16:07:57.229702 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:57.229673    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"]
Sep 17 16:07:57.231776 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:07:57.231749    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7582e2b0_c40b_4305_bd4b_23e66cb62f57.slice/crio-63cade5a4cf8c637c97a7faa345507f81ae49c594f7820fc165b660ce9e7ed14 WatchSource:0}: Error finding container 63cade5a4cf8c637c97a7faa345507f81ae49c594f7820fc165b660ce9e7ed14: Status 404 returned error can't find the container with id 63cade5a4cf8c637c97a7faa345507f81ae49c594f7820fc165b660ce9e7ed14
Sep 17 16:07:57.640943 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:57.640897    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv" event={"ID":"7582e2b0-c40b-4305-bd4b-23e66cb62f57","Type":"ContainerStarted","Data":"22c6aea7f09fa526a3c36c1bb39b61943ac4784d72e0093ece9bb3cfc62d8b7f"}
Sep 17 16:07:57.641457 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:57.641006    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv" event={"ID":"7582e2b0-c40b-4305-bd4b-23e66cb62f57","Type":"ContainerStarted","Data":"63cade5a4cf8c637c97a7faa345507f81ae49c594f7820fc165b660ce9e7ed14"}
Sep 17 16:07:57.643008 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:57.642965    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x" event={"ID":"9189f04f-771a-4b3d-bff6-e1b8db10b4d5","Type":"ContainerStarted","Data":"46ae47d523c168d119c059c90a383f497ca633c8bbf6bfa1ca7a0039230463d6"}
Sep 17 16:07:57.643154 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:57.643033    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x" event={"ID":"9189f04f-771a-4b3d-bff6-e1b8db10b4d5","Type":"ContainerStarted","Data":"34df4daf2729e84c931b345b057a39e1a940488f99885d01b8c80c4a6cef847b"}
Sep 17 16:07:57.661999 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:57.661934    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv" podStartSLOduration=1.66191603 podStartE2EDuration="1.66191603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:57.660547341 +0000 UTC m=+1743.132387188" watchObservedRunningTime="2026-09-17 16:07:57.66191603 +0000 UTC m=+1743.133755875"
Sep 17 16:07:57.680174 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:57.680115    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x" podStartSLOduration=1.680092946 podStartE2EDuration="1.680092946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:57.679008749 +0000 UTC m=+1743.150848596" watchObservedRunningTime="2026-09-17 16:07:57.680092946 +0000 UTC m=+1743.151932792"
Sep 17 16:07:57.702750 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:57.702719    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:07:58.648671 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:58.648633    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"
Sep 17 16:07:59.823431 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:07:59.823388    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:00.994700 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:00.994650    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 16:08:01.972158 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:01.972124    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:05.866709 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:05.866662    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:05.983893 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:05.983852    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:06.702588 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:06.702551    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:08:06.720887 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:06.720847    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:08:07.706046 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:07.706017    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:08:11.004611 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:11.004576    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:08:11.012845 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:11.012817    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:08:15.524377 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:15.524337    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:15.545781 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:15.545743    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"]
Sep 17 16:08:15.616397 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:15.616354    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:19.042245 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:19.042213    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:19.126819 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:19.126779    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:29.655407 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:29.655373    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"
Sep 17 16:08:30.568541 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:30.568490    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main" containerID="cri-o://919b516d201af99df43e5a5393c6b4593d12a6f0747bb315bfa918c550fc1de1" gracePeriod=45
Sep 17 16:08:31.005566 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.005517    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 16:08:31.325723 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.325695    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:08:31.402463 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.402403    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-kserve-provision-location\") pod \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") "
Sep 17 16:08:31.402650 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.402477    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-dshm\") pod \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") "
Sep 17 16:08:31.402650 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.402505    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-model-cache\") pod \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") "
Sep 17 16:08:31.402650 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.402525    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-tmp-dir\") pod \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") "
Sep 17 16:08:31.402650 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.402579    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-tls-certs\") pod \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") "
Sep 17 16:08:31.402650 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.402604    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-home\") pod \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") "
Sep 17 16:08:31.402650 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.402626    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-cabundle-cert\") pod \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") "
Sep 17 16:08:31.402650 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.402652    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtrpv\" (UniqueName: \"kubernetes.io/projected/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-kube-api-access-wtrpv\") pod \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\" (UID: \"270dce6e-6415-45aa-99dd-e9ded1e8a0ac\") "
Sep 17 16:08:31.403031 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.402823    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "270dce6e-6415-45aa-99dd-e9ded1e8a0ac" (UID: "270dce6e-6415-45aa-99dd-e9ded1e8a0ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:31.403031 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.403019    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:08:31.403148 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.403127    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "270dce6e-6415-45aa-99dd-e9ded1e8a0ac" (UID: "270dce6e-6415-45aa-99dd-e9ded1e8a0ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:31.403543 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.403515    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-home" (OuterVolumeSpecName: "home") pod "270dce6e-6415-45aa-99dd-e9ded1e8a0ac" (UID: "270dce6e-6415-45aa-99dd-e9ded1e8a0ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:31.405420 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.405393    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-dshm" (OuterVolumeSpecName: "dshm") pod "270dce6e-6415-45aa-99dd-e9ded1e8a0ac" (UID: "270dce6e-6415-45aa-99dd-e9ded1e8a0ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:31.405420 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.405400    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-kube-api-access-wtrpv" (OuterVolumeSpecName: "kube-api-access-wtrpv") pod "270dce6e-6415-45aa-99dd-e9ded1e8a0ac" (UID: "270dce6e-6415-45aa-99dd-e9ded1e8a0ac"). InnerVolumeSpecName "kube-api-access-wtrpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:08:31.405585 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.405501    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "270dce6e-6415-45aa-99dd-e9ded1e8a0ac" (UID: "270dce6e-6415-45aa-99dd-e9ded1e8a0ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:08:31.416041 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.416006    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "270dce6e-6415-45aa-99dd-e9ded1e8a0ac" (UID: "270dce6e-6415-45aa-99dd-e9ded1e8a0ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:31.427043 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.427014    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "270dce6e-6415-45aa-99dd-e9ded1e8a0ac" (UID: "270dce6e-6415-45aa-99dd-e9ded1e8a0ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:31.504299 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.504264    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:08:31.504299 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.504294    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:08:31.504299 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.504306    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:08:31.504531 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.504314    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:08:31.504531 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.504323    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:08:31.504531 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.504331    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:08:31.504531 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.504339    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtrpv\" (UniqueName: \"kubernetes.io/projected/270dce6e-6415-45aa-99dd-e9ded1e8a0ac-kube-api-access-wtrpv\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:08:31.570878 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.570792    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:31.584982 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.584942    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"]
Sep 17 16:08:31.594516 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:08:31.594474    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:08:31.594516 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:08:31.594516 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:08:31.594516 ip-10-0-129-37 kubenswrapper[2574]:  > containerID="46ae47d523c168d119c059c90a383f497ca633c8bbf6bfa1ca7a0039230463d6" cmd=["/bin/sleep","15"]
Sep 17 16:08:31.595020 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.594534    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x" message="" err=<
Sep 17 16:08:31.595020 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:08:31.595020 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:08:31.595020 ip-10-0-129-37 kubenswrapper[2574]:  >
Sep 17 16:08:31.595020 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:08:31.594552    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:08:31.595020 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:08:31.595020 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:08:31.595020 ip-10-0-129-37 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x" podUID="9189f04f-771a-4b3d-bff6-e1b8db10b4d5" containerName="main" containerID="cri-o://46ae47d523c168d119c059c90a383f497ca633c8bbf6bfa1ca7a0039230463d6"
Sep 17 16:08:31.595020 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.594600    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x" podUID="9189f04f-771a-4b3d-bff6-e1b8db10b4d5" containerName="main" containerID="cri-o://46ae47d523c168d119c059c90a383f497ca633c8bbf6bfa1ca7a0039230463d6" gracePeriod=60
Sep 17 16:08:31.599172 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.599145    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"]
Sep 17 16:08:31.663857 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.663812    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:31.694818 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.694777    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x" podUID="9189f04f-771a-4b3d-bff6-e1b8db10b4d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 16:08:31.767023 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.766979    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:31.822307 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.822222    2574 generic.go:356] "Generic (PLEG): container finished" podID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerID="919b516d201af99df43e5a5393c6b4593d12a6f0747bb315bfa918c550fc1de1" exitCode=0
Sep 17 16:08:31.822307 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.822278    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"
Sep 17 16:08:31.822612 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.822308    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" event={"ID":"270dce6e-6415-45aa-99dd-e9ded1e8a0ac","Type":"ContainerDied","Data":"919b516d201af99df43e5a5393c6b4593d12a6f0747bb315bfa918c550fc1de1"}
Sep 17 16:08:31.822612 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.822376    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm" event={"ID":"270dce6e-6415-45aa-99dd-e9ded1e8a0ac","Type":"ContainerDied","Data":"3485ef3bf2acdf4c819bd155a5dda8ab65a3812709947f15499d3a3a58b80fb6"}
Sep 17 16:08:31.822612 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.822406    2574 scope.go:117] "RemoveContainer" containerID="919b516d201af99df43e5a5393c6b4593d12a6f0747bb315bfa918c550fc1de1"
Sep 17 16:08:31.824241 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.824218    2574 generic.go:356] "Generic (PLEG): container finished" podID="9189f04f-771a-4b3d-bff6-e1b8db10b4d5" containerID="46ae47d523c168d119c059c90a383f497ca633c8bbf6bfa1ca7a0039230463d6" exitCode=0
Sep 17 16:08:31.824333 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.824297    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x" event={"ID":"9189f04f-771a-4b3d-bff6-e1b8db10b4d5","Type":"ContainerDied","Data":"46ae47d523c168d119c059c90a383f497ca633c8bbf6bfa1ca7a0039230463d6"}
Sep 17 16:08:31.833318 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.833299    2574 scope.go:117] "RemoveContainer" containerID="781636230c8b04fad5f63bcd681471e1f05368f5b90d8253fb2fe5838a63420d"
Sep 17 16:08:31.841387 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.841361    2574 scope.go:117] "RemoveContainer" containerID="919b516d201af99df43e5a5393c6b4593d12a6f0747bb315bfa918c550fc1de1"
Sep 17 16:08:31.841719 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:08:31.841699    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919b516d201af99df43e5a5393c6b4593d12a6f0747bb315bfa918c550fc1de1\": container with ID starting with 919b516d201af99df43e5a5393c6b4593d12a6f0747bb315bfa918c550fc1de1 not found: ID does not exist" containerID="919b516d201af99df43e5a5393c6b4593d12a6f0747bb315bfa918c550fc1de1"
Sep 17 16:08:31.841779 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.841730    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919b516d201af99df43e5a5393c6b4593d12a6f0747bb315bfa918c550fc1de1"} err="failed to get container status \"919b516d201af99df43e5a5393c6b4593d12a6f0747bb315bfa918c550fc1de1\": rpc error: code = NotFound desc = could not find container \"919b516d201af99df43e5a5393c6b4593d12a6f0747bb315bfa918c550fc1de1\": container with ID starting with 919b516d201af99df43e5a5393c6b4593d12a6f0747bb315bfa918c550fc1de1 not found: ID does not exist"
Sep 17 16:08:31.841779 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.841750    2574 scope.go:117] "RemoveContainer" containerID="781636230c8b04fad5f63bcd681471e1f05368f5b90d8253fb2fe5838a63420d"
Sep 17 16:08:31.841988 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:08:31.841974    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781636230c8b04fad5f63bcd681471e1f05368f5b90d8253fb2fe5838a63420d\": container with ID starting with 781636230c8b04fad5f63bcd681471e1f05368f5b90d8253fb2fe5838a63420d not found: ID does not exist" containerID="781636230c8b04fad5f63bcd681471e1f05368f5b90d8253fb2fe5838a63420d"
Sep 17 16:08:31.842027 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.841992    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781636230c8b04fad5f63bcd681471e1f05368f5b90d8253fb2fe5838a63420d"} err="failed to get container status \"781636230c8b04fad5f63bcd681471e1f05368f5b90d8253fb2fe5838a63420d\": rpc error: code = NotFound desc = could not find container \"781636230c8b04fad5f63bcd681471e1f05368f5b90d8253fb2fe5838a63420d\": container with ID starting with 781636230c8b04fad5f63bcd681471e1f05368f5b90d8253fb2fe5838a63420d not found: ID does not exist"
Sep 17 16:08:31.849367 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.849336    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"]
Sep 17 16:08:31.854748 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:31.854726    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6d9dcb8b44-vzwjm"]
Sep 17 16:08:32.346730 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.346704    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:08:32.412084 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.412001    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-dshm\") pod \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") "
Sep 17 16:08:32.412084 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.412043    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-model-cache\") pod \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") "
Sep 17 16:08:32.412084 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.412076    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-tls-certs\") pod \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") "
Sep 17 16:08:32.412394 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.412144    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-home\") pod \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") "
Sep 17 16:08:32.412394 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.412193    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-tmp-dir\") pod \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") "
Sep 17 16:08:32.412394 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.412242    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsm4l\" (UniqueName: \"kubernetes.io/projected/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-kube-api-access-tsm4l\") pod \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\" (UID: \"9189f04f-771a-4b3d-bff6-e1b8db10b4d5\") "
Sep 17 16:08:32.412394 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.412362    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "9189f04f-771a-4b3d-bff6-e1b8db10b4d5" (UID: "9189f04f-771a-4b3d-bff6-e1b8db10b4d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:32.412610 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.412461    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-home" (OuterVolumeSpecName: "home") pod "9189f04f-771a-4b3d-bff6-e1b8db10b4d5" (UID: "9189f04f-771a-4b3d-bff6-e1b8db10b4d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:32.412671 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.412625    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:08:32.412723 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.412683    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:08:32.412723 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.412698    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9189f04f-771a-4b3d-bff6-e1b8db10b4d5" (UID: "9189f04f-771a-4b3d-bff6-e1b8db10b4d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:32.414542 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.414517    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-dshm" (OuterVolumeSpecName: "dshm") pod "9189f04f-771a-4b3d-bff6-e1b8db10b4d5" (UID: "9189f04f-771a-4b3d-bff6-e1b8db10b4d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:32.414662 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.414573    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-kube-api-access-tsm4l" (OuterVolumeSpecName: "kube-api-access-tsm4l") pod "9189f04f-771a-4b3d-bff6-e1b8db10b4d5" (UID: "9189f04f-771a-4b3d-bff6-e1b8db10b4d5"). InnerVolumeSpecName "kube-api-access-tsm4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:08:32.414769 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.414750    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9189f04f-771a-4b3d-bff6-e1b8db10b4d5" (UID: "9189f04f-771a-4b3d-bff6-e1b8db10b4d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:08:32.513264 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.513231    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:08:32.513264 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.513261    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:08:32.513264 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.513272    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:08:32.513534 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.513280    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsm4l\" (UniqueName: \"kubernetes.io/projected/9189f04f-771a-4b3d-bff6-e1b8db10b4d5-kube-api-access-tsm4l\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:08:32.831619 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.831540    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"
Sep 17 16:08:32.831619 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.831578    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x" event={"ID":"9189f04f-771a-4b3d-bff6-e1b8db10b4d5","Type":"ContainerDied","Data":"34df4daf2729e84c931b345b057a39e1a940488f99885d01b8c80c4a6cef847b"}
Sep 17 16:08:32.831799 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.831669    2574 scope.go:117] "RemoveContainer" containerID="46ae47d523c168d119c059c90a383f497ca633c8bbf6bfa1ca7a0039230463d6"
Sep 17 16:08:32.862609 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.862572    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"]
Sep 17 16:08:32.867058 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:32.867034    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-b7k9x"]
Sep 17 16:08:33.048682 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:33.048650    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" path="/var/lib/kubelet/pods/270dce6e-6415-45aa-99dd-e9ded1e8a0ac/volumes"
Sep 17 16:08:33.049113 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:33.049099    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9189f04f-771a-4b3d-bff6-e1b8db10b4d5" path="/var/lib/kubelet/pods/9189f04f-771a-4b3d-bff6-e1b8db10b4d5/volumes"
Sep 17 16:08:39.655489 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:39.655433    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv" podUID="7582e2b0-c40b-4305-bd4b-23e66cb62f57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:08:46.618331 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:46.618291    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv" podUID="7582e2b0-c40b-4305-bd4b-23e66cb62f57" containerName="main" containerID="cri-o://22c6aea7f09fa526a3c36c1bb39b61943ac4784d72e0093ece9bb3cfc62d8b7f" gracePeriod=45
Sep 17 16:08:49.655265 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:49.655219    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv" podUID="7582e2b0-c40b-4305-bd4b-23e66cb62f57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:08:56.328276 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:56.328230    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:59.655624 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:59.655582    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv" podUID="7582e2b0-c40b-4305-bd4b-23e66cb62f57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:08:59.656014 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:08:59.655712    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"
Sep 17 16:09:05.578583 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:05.578543    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:05.669739 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:05.669695    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:05.879232 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:05.879201    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:05.975207 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:05.975167    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:09.655853 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:09.655814    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv" podUID="7582e2b0-c40b-4305-bd4b-23e66cb62f57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:09:10.227402 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:10.227366    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:12.440479 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:12.440427    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:14.484317 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:14.484271    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:16.866604 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:16.866575    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"
Sep 17 16:09:17.001932 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.001843    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7582e2b0-c40b-4305-bd4b-23e66cb62f57-tls-certs\") pod \"7582e2b0-c40b-4305-bd4b-23e66cb62f57\" (UID: \"7582e2b0-c40b-4305-bd4b-23e66cb62f57\") "
Sep 17 16:09:17.001932 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.001898    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssdxr\" (UniqueName: \"kubernetes.io/projected/7582e2b0-c40b-4305-bd4b-23e66cb62f57-kube-api-access-ssdxr\") pod \"7582e2b0-c40b-4305-bd4b-23e66cb62f57\" (UID: \"7582e2b0-c40b-4305-bd4b-23e66cb62f57\") "
Sep 17 16:09:17.004032 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.004000    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7582e2b0-c40b-4305-bd4b-23e66cb62f57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7582e2b0-c40b-4305-bd4b-23e66cb62f57" (UID: "7582e2b0-c40b-4305-bd4b-23e66cb62f57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:09:17.004151 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.004097    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7582e2b0-c40b-4305-bd4b-23e66cb62f57-kube-api-access-ssdxr" (OuterVolumeSpecName: "kube-api-access-ssdxr") pod "7582e2b0-c40b-4305-bd4b-23e66cb62f57" (UID: "7582e2b0-c40b-4305-bd4b-23e66cb62f57"). InnerVolumeSpecName "kube-api-access-ssdxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:09:17.047795 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.047760    2574 generic.go:356] "Generic (PLEG): container finished" podID="7582e2b0-c40b-4305-bd4b-23e66cb62f57" containerID="22c6aea7f09fa526a3c36c1bb39b61943ac4784d72e0093ece9bb3cfc62d8b7f" exitCode=0
Sep 17 16:09:17.047939 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.047870    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"
Sep 17 16:09:17.048203 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.048179    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv" event={"ID":"7582e2b0-c40b-4305-bd4b-23e66cb62f57","Type":"ContainerDied","Data":"22c6aea7f09fa526a3c36c1bb39b61943ac4784d72e0093ece9bb3cfc62d8b7f"}
Sep 17 16:09:17.048297 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.048247    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv" event={"ID":"7582e2b0-c40b-4305-bd4b-23e66cb62f57","Type":"ContainerDied","Data":"63cade5a4cf8c637c97a7faa345507f81ae49c594f7820fc165b660ce9e7ed14"}
Sep 17 16:09:17.048362 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.048317    2574 scope.go:117] "RemoveContainer" containerID="22c6aea7f09fa526a3c36c1bb39b61943ac4784d72e0093ece9bb3cfc62d8b7f"
Sep 17 16:09:17.057062 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.057044    2574 scope.go:117] "RemoveContainer" containerID="22c6aea7f09fa526a3c36c1bb39b61943ac4784d72e0093ece9bb3cfc62d8b7f"
Sep 17 16:09:17.057417 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:09:17.057343    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c6aea7f09fa526a3c36c1bb39b61943ac4784d72e0093ece9bb3cfc62d8b7f\": container with ID starting with 22c6aea7f09fa526a3c36c1bb39b61943ac4784d72e0093ece9bb3cfc62d8b7f not found: ID does not exist" containerID="22c6aea7f09fa526a3c36c1bb39b61943ac4784d72e0093ece9bb3cfc62d8b7f"
Sep 17 16:09:17.057417 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.057380    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c6aea7f09fa526a3c36c1bb39b61943ac4784d72e0093ece9bb3cfc62d8b7f"} err="failed to get container status \"22c6aea7f09fa526a3c36c1bb39b61943ac4784d72e0093ece9bb3cfc62d8b7f\": rpc error: code = NotFound desc = could not find container \"22c6aea7f09fa526a3c36c1bb39b61943ac4784d72e0093ece9bb3cfc62d8b7f\": container with ID starting with 22c6aea7f09fa526a3c36c1bb39b61943ac4784d72e0093ece9bb3cfc62d8b7f not found: ID does not exist"
Sep 17 16:09:17.074757 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.074728    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"]
Sep 17 16:09:17.078428 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.078404    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexjtfv"]
Sep 17 16:09:17.102846 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.102816    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssdxr\" (UniqueName: \"kubernetes.io/projected/7582e2b0-c40b-4305-bd4b-23e66cb62f57-kube-api-access-ssdxr\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:09:17.102846 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:17.102846    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7582e2b0-c40b-4305-bd4b-23e66cb62f57-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:09:19.049206 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:19.049167    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7582e2b0-c40b-4305-bd4b-23e66cb62f57" path="/var/lib/kubelet/pods/7582e2b0-c40b-4305-bd4b-23e66cb62f57/volumes"
Sep 17 16:09:19.061127 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:19.061094    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:19.165774 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:19.165725    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:20.925897 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:20.925849    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:21.051026 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:21.050988    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:28.055285 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:28.055232    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:34.426254 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:34.426203    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:34.528012 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:34.527976    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:40.067972 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.067939    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"]
Sep 17 16:09:40.068370 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.068355    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7582e2b0-c40b-4305-bd4b-23e66cb62f57" containerName="main"
Sep 17 16:09:40.068417 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.068374    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7582e2b0-c40b-4305-bd4b-23e66cb62f57" containerName="main"
Sep 17 16:09:40.068417 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.068400    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="storage-initializer"
Sep 17 16:09:40.068417 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.068408    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="storage-initializer"
Sep 17 16:09:40.068417 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.068416    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9189f04f-771a-4b3d-bff6-e1b8db10b4d5" containerName="main"
Sep 17 16:09:40.068569 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.068422    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9189f04f-771a-4b3d-bff6-e1b8db10b4d5" containerName="main"
Sep 17 16:09:40.068569 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.068433    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main"
Sep 17 16:09:40.068569 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.068457    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main"
Sep 17 16:09:40.068569 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.068518    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9189f04f-771a-4b3d-bff6-e1b8db10b4d5" containerName="main"
Sep 17 16:09:40.068569 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.068529    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="270dce6e-6415-45aa-99dd-e9ded1e8a0ac" containerName="main"
Sep 17 16:09:40.068569 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.068539    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7582e2b0-c40b-4305-bd4b-23e66cb62f57" containerName="main"
Sep 17 16:09:40.071746 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.071725    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.073910 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.073888    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 16:09:40.074831 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.074807    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-r8h7d\""
Sep 17 16:09:40.075076 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.075049    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 16:09:40.075196 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.075169    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 16:09:40.081262 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.081230    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"]
Sep 17 16:09:40.101103 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.101072    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:40.207088 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.207048    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcvvv\" (UniqueName: \"kubernetes.io/projected/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-kube-api-access-zcvvv\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.207281 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.207096    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.207281 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.207190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.207281 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.207232    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.207428 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.207330    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.207428 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.207397    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.308518 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.308437    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.308715 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.308562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.308715 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.308620    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcvvv\" (UniqueName: \"kubernetes.io/projected/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-kube-api-access-zcvvv\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.308715 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.308654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.308715 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.308688    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.308715 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.308713    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.308994 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.308799    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.309048 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.309021    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.309108 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.309046    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.310902 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.310872    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.311346 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.311302    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.319218 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.319137    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcvvv\" (UniqueName: \"kubernetes.io/projected/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-kube-api-access-zcvvv\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.338485 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.338460    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"]
Sep 17 16:09:40.342346 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.342324    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"
Sep 17 16:09:40.344543 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.344518    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-2v7l9\""
Sep 17 16:09:40.356756 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.356729    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"]
Sep 17 16:09:40.368416 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.368377    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:40.383623 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.383591    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:40.510580 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.510543    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldlkb\" (UniqueName: \"kubernetes.io/projected/43c8258e-c865-47c3-93a1-e820ca3082b7-kube-api-access-ldlkb\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x\" (UID: \"43c8258e-c865-47c3-93a1-e820ca3082b7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"
Sep 17 16:09:40.510580 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.510583    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c8258e-c865-47c3-93a1-e820ca3082b7-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x\" (UID: \"43c8258e-c865-47c3-93a1-e820ca3082b7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"
Sep 17 16:09:40.611269 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.611228    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldlkb\" (UniqueName: \"kubernetes.io/projected/43c8258e-c865-47c3-93a1-e820ca3082b7-kube-api-access-ldlkb\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x\" (UID: \"43c8258e-c865-47c3-93a1-e820ca3082b7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"
Sep 17 16:09:40.611431 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.611285    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c8258e-c865-47c3-93a1-e820ca3082b7-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x\" (UID: \"43c8258e-c865-47c3-93a1-e820ca3082b7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"
Sep 17 16:09:40.613958 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.613932    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c8258e-c865-47c3-93a1-e820ca3082b7-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x\" (UID: \"43c8258e-c865-47c3-93a1-e820ca3082b7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"
Sep 17 16:09:40.619757 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.619731    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldlkb\" (UniqueName: \"kubernetes.io/projected/43c8258e-c865-47c3-93a1-e820ca3082b7-kube-api-access-ldlkb\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x\" (UID: \"43c8258e-c865-47c3-93a1-e820ca3082b7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"
Sep 17 16:09:40.653105 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.653069    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"
Sep 17 16:09:40.664419 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.658902    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"]
Sep 17 16:09:40.667077 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:09:40.667048    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e418fe_5c35_40f3_b53a_61c8fea3cf1a.slice/crio-6ad1cc13cbe2fba1d261b588b2010073a05ac0468d20ae110cbe19254604dde4 WatchSource:0}: Error finding container 6ad1cc13cbe2fba1d261b588b2010073a05ac0468d20ae110cbe19254604dde4: Status 404 returned error can't find the container with id 6ad1cc13cbe2fba1d261b588b2010073a05ac0468d20ae110cbe19254604dde4
Sep 17 16:09:40.944307 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:40.944169    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"]
Sep 17 16:09:40.947018 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:09:40.946986    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c8258e_c865_47c3_93a1_e820ca3082b7.slice/crio-3b16cda45b32f784ac74ca51580b2e6c4d4bbcf93608df4fc5c2d788268d1adc WatchSource:0}: Error finding container 3b16cda45b32f784ac74ca51580b2e6c4d4bbcf93608df4fc5c2d788268d1adc: Status 404 returned error can't find the container with id 3b16cda45b32f784ac74ca51580b2e6c4d4bbcf93608df4fc5c2d788268d1adc
Sep 17 16:09:41.167246 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:41.167207    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x" event={"ID":"43c8258e-c865-47c3-93a1-e820ca3082b7","Type":"ContainerStarted","Data":"74907a6da6ec34aa034ae9cc83fb7397ef00dbbad4f71fe98740e34620cfabc4"}
Sep 17 16:09:41.167718 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:41.167269    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x" event={"ID":"43c8258e-c865-47c3-93a1-e820ca3082b7","Type":"ContainerStarted","Data":"3b16cda45b32f784ac74ca51580b2e6c4d4bbcf93608df4fc5c2d788268d1adc"}
Sep 17 16:09:41.167718 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:41.167326    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"
Sep 17 16:09:41.168825 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:41.168796    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf" event={"ID":"48e418fe-5c35-40f3-b53a-61c8fea3cf1a","Type":"ContainerStarted","Data":"7e24d57473d9bf1513f41ce64e7f142155353cf2bf5848c13e994a809c8e09a9"}
Sep 17 16:09:41.168982 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:41.168835    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf" event={"ID":"48e418fe-5c35-40f3-b53a-61c8fea3cf1a","Type":"ContainerStarted","Data":"6ad1cc13cbe2fba1d261b588b2010073a05ac0468d20ae110cbe19254604dde4"}
Sep 17 16:09:41.187573 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:41.187530    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x" podStartSLOduration=1.187515511 podStartE2EDuration="1.187515511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:09:41.186112197 +0000 UTC m=+1846.657952045" watchObservedRunningTime="2026-09-17 16:09:41.187515511 +0000 UTC m=+1846.659355357"
Sep 17 16:09:41.210817 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:41.210722    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf" podStartSLOduration=1.210706015 podStartE2EDuration="1.210706015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:09:41.208878034 +0000 UTC m=+1846.680717893" watchObservedRunningTime="2026-09-17 16:09:41.210706015 +0000 UTC m=+1846.682545882"
Sep 17 16:09:41.384141 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:41.384110    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:50.384062 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:50.384022    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:50.396280 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:50.396251    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:51.228267 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:51.228237    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:09:53.857851 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:09:53.857815    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:12.175493 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:12.175427    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"
Sep 17 16:10:27.103242 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:27.103110    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:27.270916 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:27.270871    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:29.627168 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:29.627135    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:29.887663 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:29.887588    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"]
Sep 17 16:10:29.892028 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:29.892009    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"
Sep 17 16:10:29.897269 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:29.897246    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-lj6tr\""
Sep 17 16:10:29.900347 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:29.900322    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 16:10:29.911262 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:29.911235    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"]
Sep 17 16:10:29.947108 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:29.947073    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:30.041838 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:30.041798    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/557bc796-8dde-439b-bf7a-f7b8518e5bd6-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf\" (UID: \"557bc796-8dde-439b-bf7a-f7b8518e5bd6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"
Sep 17 16:10:30.042022 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:30.041872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llcbs\" (UniqueName: \"kubernetes.io/projected/557bc796-8dde-439b-bf7a-f7b8518e5bd6-kube-api-access-llcbs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf\" (UID: \"557bc796-8dde-439b-bf7a-f7b8518e5bd6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"
Sep 17 16:10:30.142816 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:30.142725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/557bc796-8dde-439b-bf7a-f7b8518e5bd6-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf\" (UID: \"557bc796-8dde-439b-bf7a-f7b8518e5bd6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"
Sep 17 16:10:30.142988 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:30.142822    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llcbs\" (UniqueName: \"kubernetes.io/projected/557bc796-8dde-439b-bf7a-f7b8518e5bd6-kube-api-access-llcbs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf\" (UID: \"557bc796-8dde-439b-bf7a-f7b8518e5bd6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"
Sep 17 16:10:30.145327 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:30.145305    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/557bc796-8dde-439b-bf7a-f7b8518e5bd6-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf\" (UID: \"557bc796-8dde-439b-bf7a-f7b8518e5bd6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"
Sep 17 16:10:30.151144 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:30.151118    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llcbs\" (UniqueName: \"kubernetes.io/projected/557bc796-8dde-439b-bf7a-f7b8518e5bd6-kube-api-access-llcbs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf\" (UID: \"557bc796-8dde-439b-bf7a-f7b8518e5bd6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"
Sep 17 16:10:30.202260 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:30.202223    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"
Sep 17 16:10:30.482146 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:30.481957    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"]
Sep 17 16:10:30.484933 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:10:30.484904    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557bc796_8dde_439b_bf7a_f7b8518e5bd6.slice/crio-fae24d963e5590b9f4381d2201ff83d8e6685768926ecbec2572d22fad3950b2 WatchSource:0}: Error finding container fae24d963e5590b9f4381d2201ff83d8e6685768926ecbec2572d22fad3950b2: Status 404 returned error can't find the container with id fae24d963e5590b9f4381d2201ff83d8e6685768926ecbec2572d22fad3950b2
Sep 17 16:10:31.414188 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:31.414147    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf" event={"ID":"557bc796-8dde-439b-bf7a-f7b8518e5bd6","Type":"ContainerStarted","Data":"f00bb174bb967985d8f6d9e42bb8230e6ca02eec10a96e626d8db6e663e1eb9e"}
Sep 17 16:10:31.414629 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:31.414206    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf" event={"ID":"557bc796-8dde-439b-bf7a-f7b8518e5bd6","Type":"ContainerStarted","Data":"fae24d963e5590b9f4381d2201ff83d8e6685768926ecbec2572d22fad3950b2"}
Sep 17 16:10:31.414629 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:31.414280    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"
Sep 17 16:10:42.106483 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:42.106430    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:44.217256 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:44.217215    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:49.462324 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:49.462284    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:51.653128 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:51.653080    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:54.510986 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:10:54.510949    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:02.421282 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:02.421249    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"
Sep 17 16:11:02.461708 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:02.461631    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf" podStartSLOduration=33.461614655 podStartE2EDuration="33.461614655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:10:31.4352502 +0000 UTC m=+1896.907090053" watchObservedRunningTime="2026-09-17 16:11:02.461614655 +0000 UTC m=+1927.933454501"
Sep 17 16:11:07.729550 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:07.729506    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:08.431184 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:08.431147    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:15.399209 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:15.399166    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:16.238628 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:16.238589    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:17.416628 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:17.416574    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:21.952172 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:21.951993    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:21.965735 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:21.965695    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"]
Sep 17 16:11:22.049416 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:22.049362    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:22.421599 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:22.421560    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf" podUID="557bc796-8dde-439b-bf7a-f7b8518e5bd6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:11:24.249343 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:24.249293    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:32.421613 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:32.421568    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf" podUID="557bc796-8dde-439b-bf7a-f7b8518e5bd6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:11:36.986968 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:36.986921    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf" podUID="557bc796-8dde-439b-bf7a-f7b8518e5bd6" containerName="main" containerID="cri-o://f00bb174bb967985d8f6d9e42bb8230e6ca02eec10a96e626d8db6e663e1eb9e" gracePeriod=45
Sep 17 16:11:37.379190 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:37.379148    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:39.329539 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:39.329497    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:39.411252 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:39.411208    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:42.421028 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:42.420987    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf" podUID="557bc796-8dde-439b-bf7a-f7b8518e5bd6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:11:42.421425 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:42.421123    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"
Sep 17 16:11:52.420947 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:11:52.420867    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf" podUID="557bc796-8dde-439b-bf7a-f7b8518e5bd6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:12:02.420844 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:02.420805    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf" podUID="557bc796-8dde-439b-bf7a-f7b8518e5bd6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:12:07.244265 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.244199    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"
Sep 17 16:12:07.308377 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.308342    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llcbs\" (UniqueName: \"kubernetes.io/projected/557bc796-8dde-439b-bf7a-f7b8518e5bd6-kube-api-access-llcbs\") pod \"557bc796-8dde-439b-bf7a-f7b8518e5bd6\" (UID: \"557bc796-8dde-439b-bf7a-f7b8518e5bd6\") "
Sep 17 16:12:07.308563 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.308399    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/557bc796-8dde-439b-bf7a-f7b8518e5bd6-tls-certs\") pod \"557bc796-8dde-439b-bf7a-f7b8518e5bd6\" (UID: \"557bc796-8dde-439b-bf7a-f7b8518e5bd6\") "
Sep 17 16:12:07.310701 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.310667    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/557bc796-8dde-439b-bf7a-f7b8518e5bd6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "557bc796-8dde-439b-bf7a-f7b8518e5bd6" (UID: "557bc796-8dde-439b-bf7a-f7b8518e5bd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:12:07.310814 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.310752    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557bc796-8dde-439b-bf7a-f7b8518e5bd6-kube-api-access-llcbs" (OuterVolumeSpecName: "kube-api-access-llcbs") pod "557bc796-8dde-439b-bf7a-f7b8518e5bd6" (UID: "557bc796-8dde-439b-bf7a-f7b8518e5bd6"). InnerVolumeSpecName "kube-api-access-llcbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:12:07.409513 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.409471    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llcbs\" (UniqueName: \"kubernetes.io/projected/557bc796-8dde-439b-bf7a-f7b8518e5bd6-kube-api-access-llcbs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:12:07.409513 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.409505    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/557bc796-8dde-439b-bf7a-f7b8518e5bd6-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:12:07.898754 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.898719    2574 generic.go:356] "Generic (PLEG): container finished" podID="557bc796-8dde-439b-bf7a-f7b8518e5bd6" containerID="f00bb174bb967985d8f6d9e42bb8230e6ca02eec10a96e626d8db6e663e1eb9e" exitCode=0
Sep 17 16:12:07.898931 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.898768    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"
Sep 17 16:12:07.898931 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.898804    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf" event={"ID":"557bc796-8dde-439b-bf7a-f7b8518e5bd6","Type":"ContainerDied","Data":"f00bb174bb967985d8f6d9e42bb8230e6ca02eec10a96e626d8db6e663e1eb9e"}
Sep 17 16:12:07.898931 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.898856    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf" event={"ID":"557bc796-8dde-439b-bf7a-f7b8518e5bd6","Type":"ContainerDied","Data":"fae24d963e5590b9f4381d2201ff83d8e6685768926ecbec2572d22fad3950b2"}
Sep 17 16:12:07.898931 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.898881    2574 scope.go:117] "RemoveContainer" containerID="f00bb174bb967985d8f6d9e42bb8230e6ca02eec10a96e626d8db6e663e1eb9e"
Sep 17 16:12:07.911479 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.911457    2574 scope.go:117] "RemoveContainer" containerID="f00bb174bb967985d8f6d9e42bb8230e6ca02eec10a96e626d8db6e663e1eb9e"
Sep 17 16:12:07.911762 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:12:07.911742    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00bb174bb967985d8f6d9e42bb8230e6ca02eec10a96e626d8db6e663e1eb9e\": container with ID starting with f00bb174bb967985d8f6d9e42bb8230e6ca02eec10a96e626d8db6e663e1eb9e not found: ID does not exist" containerID="f00bb174bb967985d8f6d9e42bb8230e6ca02eec10a96e626d8db6e663e1eb9e"
Sep 17 16:12:07.911823 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.911770    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00bb174bb967985d8f6d9e42bb8230e6ca02eec10a96e626d8db6e663e1eb9e"} err="failed to get container status \"f00bb174bb967985d8f6d9e42bb8230e6ca02eec10a96e626d8db6e663e1eb9e\": rpc error: code = NotFound desc = could not find container \"f00bb174bb967985d8f6d9e42bb8230e6ca02eec10a96e626d8db6e663e1eb9e\": container with ID starting with f00bb174bb967985d8f6d9e42bb8230e6ca02eec10a96e626d8db6e663e1eb9e not found: ID does not exist"
Sep 17 16:12:07.924158 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.924133    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"]
Sep 17 16:12:07.930502 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:07.930481    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schewfgzf"]
Sep 17 16:12:09.048935 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:09.048898    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="557bc796-8dde-439b-bf7a-f7b8518e5bd6" path="/var/lib/kubelet/pods/557bc796-8dde-439b-bf7a-f7b8518e5bd6/volumes"
Sep 17 16:12:10.421048 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:10.420999    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"]
Sep 17 16:12:10.423847 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:10.423817    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"]
Sep 17 16:12:10.426498 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:10.426426    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:10.432636 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:12:10.432599    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:12:10.432636 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:12:10.432636 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:12:10.432636 ip-10-0-129-37 kubenswrapper[2574]:  > containerID="7e24d57473d9bf1513f41ce64e7f142155353cf2bf5848c13e994a809c8e09a9" cmd=["/bin/sleep","15"]
Sep 17 16:12:10.432923 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:10.432662    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf" message="" err=<
Sep 17 16:12:10.432923 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:12:10.432923 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:12:10.432923 ip-10-0-129-37 kubenswrapper[2574]:  >
Sep 17 16:12:10.432923 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:12:10.432683    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:12:10.432923 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:12:10.432923 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:12:10.432923 ip-10-0-129-37 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf" podUID="48e418fe-5c35-40f3-b53a-61c8fea3cf1a" containerName="main" containerID="cri-o://7e24d57473d9bf1513f41ce64e7f142155353cf2bf5848c13e994a809c8e09a9"
Sep 17 16:12:10.432923 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:10.432724    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf" podUID="48e418fe-5c35-40f3-b53a-61c8fea3cf1a" containerName="main" containerID="cri-o://7e24d57473d9bf1513f41ce64e7f142155353cf2bf5848c13e994a809c8e09a9" gracePeriod=60
Sep 17 16:12:10.516170 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:10.516136    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:10.917570 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:10.917536    2574 generic.go:356] "Generic (PLEG): container finished" podID="48e418fe-5c35-40f3-b53a-61c8fea3cf1a" containerID="7e24d57473d9bf1513f41ce64e7f142155353cf2bf5848c13e994a809c8e09a9" exitCode=0
Sep 17 16:12:10.917737 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:10.917610    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf" event={"ID":"48e418fe-5c35-40f3-b53a-61c8fea3cf1a","Type":"ContainerDied","Data":"7e24d57473d9bf1513f41ce64e7f142155353cf2bf5848c13e994a809c8e09a9"}
Sep 17 16:12:11.189328 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.189301    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:12:11.350121 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.350082    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-tmp-dir\") pod \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") "
Sep 17 16:12:11.350287 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.350142    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-model-cache\") pod \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") "
Sep 17 16:12:11.350287 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.350195    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-home\") pod \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") "
Sep 17 16:12:11.350287 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.350221    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcvvv\" (UniqueName: \"kubernetes.io/projected/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-kube-api-access-zcvvv\") pod \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") "
Sep 17 16:12:11.350287 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.350251    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-tls-certs\") pod \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") "
Sep 17 16:12:11.350287 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.350266    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-dshm\") pod \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\" (UID: \"48e418fe-5c35-40f3-b53a-61c8fea3cf1a\") "
Sep 17 16:12:11.350570 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.350407    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48e418fe-5c35-40f3-b53a-61c8fea3cf1a" (UID: "48e418fe-5c35-40f3-b53a-61c8fea3cf1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:12:11.350570 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.350433    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-model-cache" (OuterVolumeSpecName: "model-cache") pod "48e418fe-5c35-40f3-b53a-61c8fea3cf1a" (UID: "48e418fe-5c35-40f3-b53a-61c8fea3cf1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:12:11.350690 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.350569    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-home" (OuterVolumeSpecName: "home") pod "48e418fe-5c35-40f3-b53a-61c8fea3cf1a" (UID: "48e418fe-5c35-40f3-b53a-61c8fea3cf1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:12:11.350690 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.350617    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:12:11.350690 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.350632    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:12:11.352479 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.352429    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-dshm" (OuterVolumeSpecName: "dshm") pod "48e418fe-5c35-40f3-b53a-61c8fea3cf1a" (UID: "48e418fe-5c35-40f3-b53a-61c8fea3cf1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:12:11.352610 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.352578    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48e418fe-5c35-40f3-b53a-61c8fea3cf1a" (UID: "48e418fe-5c35-40f3-b53a-61c8fea3cf1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:12:11.352659 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.352606    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-kube-api-access-zcvvv" (OuterVolumeSpecName: "kube-api-access-zcvvv") pod "48e418fe-5c35-40f3-b53a-61c8fea3cf1a" (UID: "48e418fe-5c35-40f3-b53a-61c8fea3cf1a"). InnerVolumeSpecName "kube-api-access-zcvvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:12:11.451713 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.451627    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:12:11.451713 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.451667    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:12:11.451713 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.451680    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:12:11.451713 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.451693    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcvvv\" (UniqueName: \"kubernetes.io/projected/48e418fe-5c35-40f3-b53a-61c8fea3cf1a-kube-api-access-zcvvv\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:12:11.925371 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.925341    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"
Sep 17 16:12:11.925567 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.925341    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf" event={"ID":"48e418fe-5c35-40f3-b53a-61c8fea3cf1a","Type":"ContainerDied","Data":"6ad1cc13cbe2fba1d261b588b2010073a05ac0468d20ae110cbe19254604dde4"}
Sep 17 16:12:11.925567 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.925487    2574 scope.go:117] "RemoveContainer" containerID="7e24d57473d9bf1513f41ce64e7f142155353cf2bf5848c13e994a809c8e09a9"
Sep 17 16:12:11.948190 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.948160    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"]
Sep 17 16:12:11.952427 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:11.952398    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7jswf"]
Sep 17 16:12:12.176059 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:12.175972    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x" podUID="43c8258e-c865-47c3-93a1-e820ca3082b7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:12:13.048460 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:13.048408    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48e418fe-5c35-40f3-b53a-61c8fea3cf1a" path="/var/lib/kubelet/pods/48e418fe-5c35-40f3-b53a-61c8fea3cf1a/volumes"
Sep 17 16:12:22.176039 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:22.175992    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x" podUID="43c8258e-c865-47c3-93a1-e820ca3082b7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:12:25.449689 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:25.449639    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x" podUID="43c8258e-c865-47c3-93a1-e820ca3082b7" containerName="main" containerID="cri-o://74907a6da6ec34aa034ae9cc83fb7397ef00dbbad4f71fe98740e34620cfabc4" gracePeriod=45
Sep 17 16:12:32.175490 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:32.175423    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x" podUID="43c8258e-c865-47c3-93a1-e820ca3082b7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:12:32.175971 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:32.175551    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"
Sep 17 16:12:42.176208 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:42.176163    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x" podUID="43c8258e-c865-47c3-93a1-e820ca3082b7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:12:52.175758 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:52.175713    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x" podUID="43c8258e-c865-47c3-93a1-e820ca3082b7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:12:55.704375 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:55.704351    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"
Sep 17 16:12:55.844559 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:55.844472    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c8258e-c865-47c3-93a1-e820ca3082b7-tls-certs\") pod \"43c8258e-c865-47c3-93a1-e820ca3082b7\" (UID: \"43c8258e-c865-47c3-93a1-e820ca3082b7\") "
Sep 17 16:12:55.844559 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:55.844510    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldlkb\" (UniqueName: \"kubernetes.io/projected/43c8258e-c865-47c3-93a1-e820ca3082b7-kube-api-access-ldlkb\") pod \"43c8258e-c865-47c3-93a1-e820ca3082b7\" (UID: \"43c8258e-c865-47c3-93a1-e820ca3082b7\") "
Sep 17 16:12:55.846580 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:55.846553    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c8258e-c865-47c3-93a1-e820ca3082b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43c8258e-c865-47c3-93a1-e820ca3082b7" (UID: "43c8258e-c865-47c3-93a1-e820ca3082b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:12:55.846715 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:55.846701    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c8258e-c865-47c3-93a1-e820ca3082b7-kube-api-access-ldlkb" (OuterVolumeSpecName: "kube-api-access-ldlkb") pod "43c8258e-c865-47c3-93a1-e820ca3082b7" (UID: "43c8258e-c865-47c3-93a1-e820ca3082b7"). InnerVolumeSpecName "kube-api-access-ldlkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:12:55.945629 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:55.945577    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c8258e-c865-47c3-93a1-e820ca3082b7-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:12:55.945629 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:55.945621    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldlkb\" (UniqueName: \"kubernetes.io/projected/43c8258e-c865-47c3-93a1-e820ca3082b7-kube-api-access-ldlkb\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:12:56.133875 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:56.133838    2574 generic.go:356] "Generic (PLEG): container finished" podID="43c8258e-c865-47c3-93a1-e820ca3082b7" containerID="74907a6da6ec34aa034ae9cc83fb7397ef00dbbad4f71fe98740e34620cfabc4" exitCode=0
Sep 17 16:12:56.134042 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:56.133878    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"
Sep 17 16:12:56.134042 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:56.133921    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x" event={"ID":"43c8258e-c865-47c3-93a1-e820ca3082b7","Type":"ContainerDied","Data":"74907a6da6ec34aa034ae9cc83fb7397ef00dbbad4f71fe98740e34620cfabc4"}
Sep 17 16:12:56.134042 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:56.133975    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x" event={"ID":"43c8258e-c865-47c3-93a1-e820ca3082b7","Type":"ContainerDied","Data":"3b16cda45b32f784ac74ca51580b2e6c4d4bbcf93608df4fc5c2d788268d1adc"}
Sep 17 16:12:56.134042 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:56.133999    2574 scope.go:117] "RemoveContainer" containerID="74907a6da6ec34aa034ae9cc83fb7397ef00dbbad4f71fe98740e34620cfabc4"
Sep 17 16:12:56.142711 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:56.142693    2574 scope.go:117] "RemoveContainer" containerID="74907a6da6ec34aa034ae9cc83fb7397ef00dbbad4f71fe98740e34620cfabc4"
Sep 17 16:12:56.143002 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:12:56.142983    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74907a6da6ec34aa034ae9cc83fb7397ef00dbbad4f71fe98740e34620cfabc4\": container with ID starting with 74907a6da6ec34aa034ae9cc83fb7397ef00dbbad4f71fe98740e34620cfabc4 not found: ID does not exist" containerID="74907a6da6ec34aa034ae9cc83fb7397ef00dbbad4f71fe98740e34620cfabc4"
Sep 17 16:12:56.143041 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:56.143013    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74907a6da6ec34aa034ae9cc83fb7397ef00dbbad4f71fe98740e34620cfabc4"} err="failed to get container status \"74907a6da6ec34aa034ae9cc83fb7397ef00dbbad4f71fe98740e34620cfabc4\": rpc error: code = NotFound desc = could not find container \"74907a6da6ec34aa034ae9cc83fb7397ef00dbbad4f71fe98740e34620cfabc4\": container with ID starting with 74907a6da6ec34aa034ae9cc83fb7397ef00dbbad4f71fe98740e34620cfabc4 not found: ID does not exist"
Sep 17 16:12:56.159712 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:56.159684    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"]
Sep 17 16:12:56.163096 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:56.163060    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schextz6x"]
Sep 17 16:12:57.049769 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:12:57.049732    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43c8258e-c865-47c3-93a1-e820ca3082b7" path="/var/lib/kubelet/pods/43c8258e-c865-47c3-93a1-e820ca3082b7/volumes"
Sep 17 16:13:00.478473 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:00.478417    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:00.575519 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:00.575478    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:07.505572 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:07.505537    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:10.084895 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.084857    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"]
Sep 17 16:13:10.085276 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.085249    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="557bc796-8dde-439b-bf7a-f7b8518e5bd6" containerName="main"
Sep 17 16:13:10.085276 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.085261    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="557bc796-8dde-439b-bf7a-f7b8518e5bd6" containerName="main"
Sep 17 16:13:10.085276 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.085270    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c8258e-c865-47c3-93a1-e820ca3082b7" containerName="main"
Sep 17 16:13:10.085276 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.085276    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c8258e-c865-47c3-93a1-e820ca3082b7" containerName="main"
Sep 17 16:13:10.085463 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.085287    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48e418fe-5c35-40f3-b53a-61c8fea3cf1a" containerName="main"
Sep 17 16:13:10.085463 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.085292    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="48e418fe-5c35-40f3-b53a-61c8fea3cf1a" containerName="main"
Sep 17 16:13:10.085463 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.085348    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="557bc796-8dde-439b-bf7a-f7b8518e5bd6" containerName="main"
Sep 17 16:13:10.085463 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.085357    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="43c8258e-c865-47c3-93a1-e820ca3082b7" containerName="main"
Sep 17 16:13:10.085463 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.085363    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="48e418fe-5c35-40f3-b53a-61c8fea3cf1a" containerName="main"
Sep 17 16:13:10.088548 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.088530    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.090863 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.090836    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-kkjx7\""
Sep 17 16:13:10.091004 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.090881    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 16:13:10.091698 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.091677    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 16:13:10.091832 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.091706    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 16:13:10.102007 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.101986    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"]
Sep 17 16:13:10.134714 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.134687    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:10.271770 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.271732    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.271952 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.271788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctxg8\" (UniqueName: \"kubernetes.io/projected/cdbce933-78dd-4209-b23d-806264e346f3-kube-api-access-ctxg8\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.271952 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.271854    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.271952 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.271876    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.271952 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.271901    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.271952 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.271921    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdbce933-78dd-4209-b23d-806264e346f3-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.296388 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.296351    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"]
Sep 17 16:13:10.300006 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.299990    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"
Sep 17 16:13:10.302860 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.302834    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-qrgdb\""
Sep 17 16:13:10.315246 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.315222    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"]
Sep 17 16:13:10.337328 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.337223    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:10.373058 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.373031    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.373210 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.373062    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.373210 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.373087    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.373210 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.373117    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdbce933-78dd-4209-b23d-806264e346f3-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.373210 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.373158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.373210 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.373205    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctxg8\" (UniqueName: \"kubernetes.io/projected/cdbce933-78dd-4209-b23d-806264e346f3-kube-api-access-ctxg8\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.373560 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.373535    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.373632 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.373563    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.373691 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.373674    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.375554 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.375524    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.375698 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.375682    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdbce933-78dd-4209-b23d-806264e346f3-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.381400 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.381369    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctxg8\" (UniqueName: \"kubernetes.io/projected/cdbce933-78dd-4209-b23d-806264e346f3-kube-api-access-ctxg8\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.399331 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.399307    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:10.474193 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.474106    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p5h4\" (UniqueName: \"kubernetes.io/projected/f56ae0a4-cbaf-4a69-953e-21eda4e76d59-kube-api-access-7p5h4\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw\" (UID: \"f56ae0a4-cbaf-4a69-953e-21eda4e76d59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"
Sep 17 16:13:10.474349 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.474220    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56ae0a4-cbaf-4a69-953e-21eda4e76d59-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw\" (UID: \"f56ae0a4-cbaf-4a69-953e-21eda4e76d59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"
Sep 17 16:13:10.575519 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.575482    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p5h4\" (UniqueName: \"kubernetes.io/projected/f56ae0a4-cbaf-4a69-953e-21eda4e76d59-kube-api-access-7p5h4\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw\" (UID: \"f56ae0a4-cbaf-4a69-953e-21eda4e76d59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"
Sep 17 16:13:10.575715 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.575529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56ae0a4-cbaf-4a69-953e-21eda4e76d59-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw\" (UID: \"f56ae0a4-cbaf-4a69-953e-21eda4e76d59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"
Sep 17 16:13:10.578348 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.578309    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56ae0a4-cbaf-4a69-953e-21eda4e76d59-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw\" (UID: \"f56ae0a4-cbaf-4a69-953e-21eda4e76d59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"
Sep 17 16:13:10.583919 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.583894    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p5h4\" (UniqueName: \"kubernetes.io/projected/f56ae0a4-cbaf-4a69-953e-21eda4e76d59-kube-api-access-7p5h4\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw\" (UID: \"f56ae0a4-cbaf-4a69-953e-21eda4e76d59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"
Sep 17 16:13:10.611768 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.611739    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"
Sep 17 16:13:10.677231 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.677198    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"]
Sep 17 16:13:10.682076 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:13:10.682049    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbce933_78dd_4209_b23d_806264e346f3.slice/crio-f13afbea448135a7f911b84c4e8bcebe347125a347cb3f82d4204c6db6bdd861 WatchSource:0}: Error finding container f13afbea448135a7f911b84c4e8bcebe347125a347cb3f82d4204c6db6bdd861: Status 404 returned error can't find the container with id f13afbea448135a7f911b84c4e8bcebe347125a347cb3f82d4204c6db6bdd861
Sep 17 16:13:10.886967 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:10.886939    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"]
Sep 17 16:13:10.888949 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:13:10.888923    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56ae0a4_cbaf_4a69_953e_21eda4e76d59.slice/crio-2a84ef4d020853666d09efb7f5bd80157401799522a959e82a6a39bd38b16fe2 WatchSource:0}: Error finding container 2a84ef4d020853666d09efb7f5bd80157401799522a959e82a6a39bd38b16fe2: Status 404 returned error can't find the container with id 2a84ef4d020853666d09efb7f5bd80157401799522a959e82a6a39bd38b16fe2
Sep 17 16:13:11.206433 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:11.206331    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw" event={"ID":"f56ae0a4-cbaf-4a69-953e-21eda4e76d59","Type":"ContainerStarted","Data":"2f2e23499426ba074ddb8bda8d41cd586647d6d517208f17e4f978bb61aa5d77"}
Sep 17 16:13:11.206848 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:11.206541    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"
Sep 17 16:13:11.206848 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:11.206580    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw" event={"ID":"f56ae0a4-cbaf-4a69-953e-21eda4e76d59","Type":"ContainerStarted","Data":"2a84ef4d020853666d09efb7f5bd80157401799522a959e82a6a39bd38b16fe2"}
Sep 17 16:13:11.207972 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:11.207945    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl" event={"ID":"cdbce933-78dd-4209-b23d-806264e346f3","Type":"ContainerStarted","Data":"ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966"}
Sep 17 16:13:11.208088 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:11.207986    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl" event={"ID":"cdbce933-78dd-4209-b23d-806264e346f3","Type":"ContainerStarted","Data":"f13afbea448135a7f911b84c4e8bcebe347125a347cb3f82d4204c6db6bdd861"}
Sep 17 16:13:11.245129 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:11.245066    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw" podStartSLOduration=1.245050754 podStartE2EDuration="1.245050754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:13:11.242345731 +0000 UTC m=+2056.714185583" watchObservedRunningTime="2026-09-17 16:13:11.245050754 +0000 UTC m=+2056.716890605"
Sep 17 16:13:11.279659 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:11.279602    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl" podStartSLOduration=1.2795867539999999 podStartE2EDuration="1.279586754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:13:11.278473347 +0000 UTC m=+2056.750313194" watchObservedRunningTime="2026-09-17 16:13:11.279586754 +0000 UTC m=+2056.751426639"
Sep 17 16:13:11.400243 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:11.400214    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:13.537979 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:13.537945    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:13.638798 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:13.638756    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:17.054946 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:17.054908    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:20.399927 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:20.399881    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:20.412571 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:20.412545    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:21.270942 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:21.270908    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:13:23.826395 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:23.826356    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:23.975359 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:23.975321    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:42.216131 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:42.216096    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"
Sep 17 16:13:52.194767 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:52.194726    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:52.539483 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:52.539365    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:59.933045 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:13:59.932999    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:13.303190 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:14:13.303144    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:13.395349 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:14:13.395305    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:20.092275 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:14:20.092229    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:26.172904 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:14:26.172870    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:26.272878 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:14:26.272809    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:31.649397 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:14:31.649341    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:00.312074 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.311994    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:00.429228 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.429193    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"]
Sep 17 16:15:00.432719 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.432695    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"
Sep 17 16:15:00.436313 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.436288    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-hjx9r\""
Sep 17 16:15:00.436471 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.436288    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-6g6k7\""
Sep 17 16:15:00.436471 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.436331    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 16:15:00.436471 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.436358    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 16:15:00.436771 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.436716    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 16:15:00.443592 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.443569    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"]
Sep 17 16:15:00.465304 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.465270    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf5f2\" (UniqueName: \"kubernetes.io/projected/065e7b92-223c-4886-879e-0dfa0c6b5a84-kube-api-access-zf5f2\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr\" (UID: \"065e7b92-223c-4886-879e-0dfa0c6b5a84\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"
Sep 17 16:15:00.465489 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.465341    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/065e7b92-223c-4886-879e-0dfa0c6b5a84-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr\" (UID: \"065e7b92-223c-4886-879e-0dfa0c6b5a84\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"
Sep 17 16:15:00.466864 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.466834    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:00.566848 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.566765    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5f2\" (UniqueName: \"kubernetes.io/projected/065e7b92-223c-4886-879e-0dfa0c6b5a84-kube-api-access-zf5f2\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr\" (UID: \"065e7b92-223c-4886-879e-0dfa0c6b5a84\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"
Sep 17 16:15:00.566848 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.566807    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/065e7b92-223c-4886-879e-0dfa0c6b5a84-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr\" (UID: \"065e7b92-223c-4886-879e-0dfa0c6b5a84\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"
Sep 17 16:15:00.569304 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.569272    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/065e7b92-223c-4886-879e-0dfa0c6b5a84-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr\" (UID: \"065e7b92-223c-4886-879e-0dfa0c6b5a84\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"
Sep 17 16:15:00.575475 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.575421    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf5f2\" (UniqueName: \"kubernetes.io/projected/065e7b92-223c-4886-879e-0dfa0c6b5a84-kube-api-access-zf5f2\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr\" (UID: \"065e7b92-223c-4886-879e-0dfa0c6b5a84\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"
Sep 17 16:15:00.744211 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:00.744169    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"
Sep 17 16:15:01.019923 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:01.019893    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"]
Sep 17 16:15:01.022282 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:15:01.022246    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065e7b92_223c_4886_879e_0dfa0c6b5a84.slice/crio-217711a0fd29bdad4fa3ec4eb1e581c13101ac2098f9ef60565e911bc4073f1f WatchSource:0}: Error finding container 217711a0fd29bdad4fa3ec4eb1e581c13101ac2098f9ef60565e911bc4073f1f: Status 404 returned error can't find the container with id 217711a0fd29bdad4fa3ec4eb1e581c13101ac2098f9ef60565e911bc4073f1f
Sep 17 16:15:01.743684 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:01.743644    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr" event={"ID":"065e7b92-223c-4886-879e-0dfa0c6b5a84","Type":"ContainerStarted","Data":"7ece17ff478a91dd9e85844a9abd3444f7c8b0936ea337a1a26188986092be8a"}
Sep 17 16:15:01.744169 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:01.743705    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr" event={"ID":"065e7b92-223c-4886-879e-0dfa0c6b5a84","Type":"ContainerStarted","Data":"217711a0fd29bdad4fa3ec4eb1e581c13101ac2098f9ef60565e911bc4073f1f"}
Sep 17 16:15:01.744169 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:01.743734    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"
Sep 17 16:15:01.769681 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:01.769628    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr" podStartSLOduration=1.769612338 podStartE2EDuration="1.769612338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:15:01.76721383 +0000 UTC m=+2167.239053698" watchObservedRunningTime="2026-09-17 16:15:01.769612338 +0000 UTC m=+2167.241452203"
Sep 17 16:15:18.756895 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:18.756852    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:26.702355 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:26.702314    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:32.750615 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:32.750585    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"
Sep 17 16:15:37.018564 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:37.018518    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:37.110834 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:15:37.110792    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:16:26.015081 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.014869    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"]
Sep 17 16:16:26.017693 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.017661    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"]
Sep 17 16:16:26.019509 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.019068    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:16:26.028481 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:16:26.028414    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:16:26.028481 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:16:26.028481 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:16:26.028481 ip-10-0-129-37 kubenswrapper[2574]:  > containerID="ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966" cmd=["/bin/sleep","15"]
Sep 17 16:16:26.028800 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.028504    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl" message="" err=<
Sep 17 16:16:26.028800 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:16:26.028800 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:16:26.028800 ip-10-0-129-37 kubenswrapper[2574]:  >
Sep 17 16:16:26.028800 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:16:26.028521    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:16:26.028800 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:16:26.028800 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:16:26.028800 ip-10-0-129-37 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl" podUID="cdbce933-78dd-4209-b23d-806264e346f3" containerName="main" containerID="cri-o://ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966"
Sep 17 16:16:26.028800 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.028584    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl" podUID="cdbce933-78dd-4209-b23d-806264e346f3" containerName="main" containerID="cri-o://ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966" gracePeriod=60
Sep 17 16:16:26.260007 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.259966    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl" podUID="cdbce933-78dd-4209-b23d-806264e346f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 16:16:26.388170 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.388141    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:16:26.439604 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.439567    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctxg8\" (UniqueName: \"kubernetes.io/projected/cdbce933-78dd-4209-b23d-806264e346f3-kube-api-access-ctxg8\") pod \"cdbce933-78dd-4209-b23d-806264e346f3\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") "
Sep 17 16:16:26.439800 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.439638    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-home\") pod \"cdbce933-78dd-4209-b23d-806264e346f3\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") "
Sep 17 16:16:26.439800 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.439695    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-model-cache\") pod \"cdbce933-78dd-4209-b23d-806264e346f3\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") "
Sep 17 16:16:26.439800 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.439763    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-dshm\") pod \"cdbce933-78dd-4209-b23d-806264e346f3\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") "
Sep 17 16:16:26.439967 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.439803    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-tmp-dir\") pod \"cdbce933-78dd-4209-b23d-806264e346f3\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") "
Sep 17 16:16:26.439967 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.439833    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdbce933-78dd-4209-b23d-806264e346f3-tls-certs\") pod \"cdbce933-78dd-4209-b23d-806264e346f3\" (UID: \"cdbce933-78dd-4209-b23d-806264e346f3\") "
Sep 17 16:16:26.439967 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.439918    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-home" (OuterVolumeSpecName: "home") pod "cdbce933-78dd-4209-b23d-806264e346f3" (UID: "cdbce933-78dd-4209-b23d-806264e346f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:16:26.440126 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.439960    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "cdbce933-78dd-4209-b23d-806264e346f3" (UID: "cdbce933-78dd-4209-b23d-806264e346f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:16:26.440126 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.440066    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cdbce933-78dd-4209-b23d-806264e346f3" (UID: "cdbce933-78dd-4209-b23d-806264e346f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:16:26.440242 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.440132    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:16:26.440242 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.440167    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:16:26.440242 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.440176    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:16:26.442033 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.442004    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-dshm" (OuterVolumeSpecName: "dshm") pod "cdbce933-78dd-4209-b23d-806264e346f3" (UID: "cdbce933-78dd-4209-b23d-806264e346f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:16:26.442145 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.442051    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdbce933-78dd-4209-b23d-806264e346f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdbce933-78dd-4209-b23d-806264e346f3" (UID: "cdbce933-78dd-4209-b23d-806264e346f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:16:26.442145 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.442081    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbce933-78dd-4209-b23d-806264e346f3-kube-api-access-ctxg8" (OuterVolumeSpecName: "kube-api-access-ctxg8") pod "cdbce933-78dd-4209-b23d-806264e346f3" (UID: "cdbce933-78dd-4209-b23d-806264e346f3"). InnerVolumeSpecName "kube-api-access-ctxg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:16:26.540954 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.540921    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdbce933-78dd-4209-b23d-806264e346f3-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:16:26.540954 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.540953    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdbce933-78dd-4209-b23d-806264e346f3-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:16:26.541156 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:26.540968    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctxg8\" (UniqueName: \"kubernetes.io/projected/cdbce933-78dd-4209-b23d-806264e346f3-kube-api-access-ctxg8\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:16:27.163364 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:27.163330    2574 generic.go:356] "Generic (PLEG): container finished" podID="cdbce933-78dd-4209-b23d-806264e346f3" containerID="ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966" exitCode=0
Sep 17 16:16:27.163364 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:27.163373    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"
Sep 17 16:16:27.163364 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:27.163402    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl" event={"ID":"cdbce933-78dd-4209-b23d-806264e346f3","Type":"ContainerDied","Data":"ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966"}
Sep 17 16:16:27.163972 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:27.163474    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl" event={"ID":"cdbce933-78dd-4209-b23d-806264e346f3","Type":"ContainerDied","Data":"f13afbea448135a7f911b84c4e8bcebe347125a347cb3f82d4204c6db6bdd861"}
Sep 17 16:16:27.163972 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:27.163508    2574 scope.go:117] "RemoveContainer" containerID="ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966"
Sep 17 16:16:27.179285 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:27.179238    2574 scope.go:117] "RemoveContainer" containerID="ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966"
Sep 17 16:16:27.179805 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:16:27.179770    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966\": container with ID starting with ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966 not found: ID does not exist" containerID="ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966"
Sep 17 16:16:27.179908 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:27.179828    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966"} err="failed to get container status \"ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966\": rpc error: code = NotFound desc = could not find container \"ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966\": container with ID starting with ddeb5a7ccb23d1ac26f25d5428375b65f65de90ac5bd2e9772dd204c5d86f966 not found: ID does not exist"
Sep 17 16:16:27.183076 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:27.183051    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"]
Sep 17 16:16:27.186582 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:27.186559    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969frxjl"]
Sep 17 16:16:27.307357 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:27.307318    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:16:29.048983 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:29.048939    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdbce933-78dd-4209-b23d-806264e346f3" path="/var/lib/kubelet/pods/cdbce933-78dd-4209-b23d-806264e346f3/volumes"
Sep 17 16:16:32.216672 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:32.216627    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw" podUID="f56ae0a4-cbaf-4a69-953e-21eda4e76d59" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:16:41.037285 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:41.037231    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw" podUID="f56ae0a4-cbaf-4a69-953e-21eda4e76d59" containerName="main" containerID="cri-o://2f2e23499426ba074ddb8bda8d41cd586647d6d517208f17e4f978bb61aa5d77" gracePeriod=45
Sep 17 16:16:42.216595 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:42.216556    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw" podUID="f56ae0a4-cbaf-4a69-953e-21eda4e76d59" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:16:52.217010 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:52.216966    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw" podUID="f56ae0a4-cbaf-4a69-953e-21eda4e76d59" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:16:52.217416 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:16:52.217089    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"
Sep 17 16:17:02.216236 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:02.216192    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw" podUID="f56ae0a4-cbaf-4a69-953e-21eda4e76d59" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:17:11.284231 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.284206    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"
Sep 17 16:17:11.382889 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.382855    2574 generic.go:356] "Generic (PLEG): container finished" podID="f56ae0a4-cbaf-4a69-953e-21eda4e76d59" containerID="2f2e23499426ba074ddb8bda8d41cd586647d6d517208f17e4f978bb61aa5d77" exitCode=0
Sep 17 16:17:11.383060 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.382903    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"
Sep 17 16:17:11.383060 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.382935    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw" event={"ID":"f56ae0a4-cbaf-4a69-953e-21eda4e76d59","Type":"ContainerDied","Data":"2f2e23499426ba074ddb8bda8d41cd586647d6d517208f17e4f978bb61aa5d77"}
Sep 17 16:17:11.383060 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.382978    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw" event={"ID":"f56ae0a4-cbaf-4a69-953e-21eda4e76d59","Type":"ContainerDied","Data":"2a84ef4d020853666d09efb7f5bd80157401799522a959e82a6a39bd38b16fe2"}
Sep 17 16:17:11.383060 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.383000    2574 scope.go:117] "RemoveContainer" containerID="2f2e23499426ba074ddb8bda8d41cd586647d6d517208f17e4f978bb61aa5d77"
Sep 17 16:17:11.391681 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.391661    2574 scope.go:117] "RemoveContainer" containerID="2f2e23499426ba074ddb8bda8d41cd586647d6d517208f17e4f978bb61aa5d77"
Sep 17 16:17:11.391947 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:17:11.391929    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f2e23499426ba074ddb8bda8d41cd586647d6d517208f17e4f978bb61aa5d77\": container with ID starting with 2f2e23499426ba074ddb8bda8d41cd586647d6d517208f17e4f978bb61aa5d77 not found: ID does not exist" containerID="2f2e23499426ba074ddb8bda8d41cd586647d6d517208f17e4f978bb61aa5d77"
Sep 17 16:17:11.391992 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.391957    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f2e23499426ba074ddb8bda8d41cd586647d6d517208f17e4f978bb61aa5d77"} err="failed to get container status \"2f2e23499426ba074ddb8bda8d41cd586647d6d517208f17e4f978bb61aa5d77\": rpc error: code = NotFound desc = could not find container \"2f2e23499426ba074ddb8bda8d41cd586647d6d517208f17e4f978bb61aa5d77\": container with ID starting with 2f2e23499426ba074ddb8bda8d41cd586647d6d517208f17e4f978bb61aa5d77 not found: ID does not exist"
Sep 17 16:17:11.431932 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.431903    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56ae0a4-cbaf-4a69-953e-21eda4e76d59-tls-certs\") pod \"f56ae0a4-cbaf-4a69-953e-21eda4e76d59\" (UID: \"f56ae0a4-cbaf-4a69-953e-21eda4e76d59\") "
Sep 17 16:17:11.432064 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.431980    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p5h4\" (UniqueName: \"kubernetes.io/projected/f56ae0a4-cbaf-4a69-953e-21eda4e76d59-kube-api-access-7p5h4\") pod \"f56ae0a4-cbaf-4a69-953e-21eda4e76d59\" (UID: \"f56ae0a4-cbaf-4a69-953e-21eda4e76d59\") "
Sep 17 16:17:11.434124 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.434092    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56ae0a4-cbaf-4a69-953e-21eda4e76d59-kube-api-access-7p5h4" (OuterVolumeSpecName: "kube-api-access-7p5h4") pod "f56ae0a4-cbaf-4a69-953e-21eda4e76d59" (UID: "f56ae0a4-cbaf-4a69-953e-21eda4e76d59"). InnerVolumeSpecName "kube-api-access-7p5h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:17:11.434262 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.434133    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f56ae0a4-cbaf-4a69-953e-21eda4e76d59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f56ae0a4-cbaf-4a69-953e-21eda4e76d59" (UID: "f56ae0a4-cbaf-4a69-953e-21eda4e76d59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:17:11.532935 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.532899    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56ae0a4-cbaf-4a69-953e-21eda4e76d59-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:17:11.532935 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.532929    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p5h4\" (UniqueName: \"kubernetes.io/projected/f56ae0a4-cbaf-4a69-953e-21eda4e76d59-kube-api-access-7p5h4\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:17:11.713913 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.713879    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"]
Sep 17 16:17:11.719430 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:11.719400    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche959jw"]
Sep 17 16:17:13.049365 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:13.049333    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f56ae0a4-cbaf-4a69-953e-21eda4e76d59" path="/var/lib/kubelet/pods/f56ae0a4-cbaf-4a69-953e-21eda4e76d59/volumes"
Sep 17 16:17:16.071164 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:16.071128    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:16.166773 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:16.166734    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:18.908833 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.908793    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"]
Sep 17 16:17:18.909287 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.909263    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f56ae0a4-cbaf-4a69-953e-21eda4e76d59" containerName="main"
Sep 17 16:17:18.909287 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.909286    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f56ae0a4-cbaf-4a69-953e-21eda4e76d59" containerName="main"
Sep 17 16:17:18.909539 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.909311    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbce933-78dd-4209-b23d-806264e346f3" containerName="main"
Sep 17 16:17:18.909539 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.909319    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbce933-78dd-4209-b23d-806264e346f3" containerName="main"
Sep 17 16:17:18.909539 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.909453    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdbce933-78dd-4209-b23d-806264e346f3" containerName="main"
Sep 17 16:17:18.909539 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.909472    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f56ae0a4-cbaf-4a69-953e-21eda4e76d59" containerName="main"
Sep 17 16:17:18.914122 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.914101    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:18.916533 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.916482    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 16:17:18.916717 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.916556    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:17:18.916717 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.916597    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 17 16:17:18.916717 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.916694    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 16:17:18.916894 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.916714    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 16:17:18.916949 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.916923    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-p8g9m\""
Sep 17 16:17:18.932825 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.932800    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"]
Sep 17 16:17:18.949954 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.949926    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:18.995986 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.995946    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:18.995986 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.995984    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acc1ca37-4dea-4944-9125-fb2f7b67f95f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:18.996197 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.996014    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:18.996197 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.996083    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acc1ca37-4dea-4944-9125-fb2f7b67f95f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:18.996197 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.996135    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:18.996197 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.996174    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:18.996366 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.996203    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b5n7\" (UniqueName: \"kubernetes.io/projected/acc1ca37-4dea-4944-9125-fb2f7b67f95f-kube-api-access-2b5n7\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:18.996366 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:18.996234    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.097228 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.097192    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acc1ca37-4dea-4944-9125-fb2f7b67f95f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.097430 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.097247    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.097430 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.097265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.097430 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.097392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b5n7\" (UniqueName: \"kubernetes.io/projected/acc1ca37-4dea-4944-9125-fb2f7b67f95f-kube-api-access-2b5n7\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.097650 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.097481    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.097650 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.097558    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.097650 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.097589    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acc1ca37-4dea-4944-9125-fb2f7b67f95f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.097650 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.097619    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.097849 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.097678    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.097849 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.097709    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.097994 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.097973    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.098048 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.097998    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.098239 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.098222    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acc1ca37-4dea-4944-9125-fb2f7b67f95f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.099707 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.099686    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.099997 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.099952    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acc1ca37-4dea-4944-9125-fb2f7b67f95f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.105795 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.105763    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b5n7\" (UniqueName: \"kubernetes.io/projected/acc1ca37-4dea-4944-9125-fb2f7b67f95f-kube-api-access-2b5n7\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.133738 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.133692    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"]
Sep 17 16:17:19.138430 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.138405    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"
Sep 17 16:17:19.140704 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.140680    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-rqcx9\""
Sep 17 16:17:19.145641 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.145616    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"]
Sep 17 16:17:19.183361 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.183275    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:19.198531 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.198498    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5hp9\" (UniqueName: \"kubernetes.io/projected/8a10d675-6269-4a2f-9665-4c83c666d18f-kube-api-access-p5hp9\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2\" (UID: \"8a10d675-6269-4a2f-9665-4c83c666d18f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"
Sep 17 16:17:19.198689 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.198605    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a10d675-6269-4a2f-9665-4c83c666d18f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2\" (UID: \"8a10d675-6269-4a2f-9665-4c83c666d18f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"
Sep 17 16:17:19.224816 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.224779    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:19.299707 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.299650    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a10d675-6269-4a2f-9665-4c83c666d18f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2\" (UID: \"8a10d675-6269-4a2f-9665-4c83c666d18f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"
Sep 17 16:17:19.299870 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.299744    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5hp9\" (UniqueName: \"kubernetes.io/projected/8a10d675-6269-4a2f-9665-4c83c666d18f-kube-api-access-p5hp9\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2\" (UID: \"8a10d675-6269-4a2f-9665-4c83c666d18f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"
Sep 17 16:17:19.301997 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.301971    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a10d675-6269-4a2f-9665-4c83c666d18f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2\" (UID: \"8a10d675-6269-4a2f-9665-4c83c666d18f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"
Sep 17 16:17:19.308151 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.308100    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5hp9\" (UniqueName: \"kubernetes.io/projected/8a10d675-6269-4a2f-9665-4c83c666d18f-kube-api-access-p5hp9\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2\" (UID: \"8a10d675-6269-4a2f-9665-4c83c666d18f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"
Sep 17 16:17:19.450064 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.449967    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"
Sep 17 16:17:19.514128 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.514095    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"]
Sep 17 16:17:19.519317 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:17:19.519283    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc1ca37_4dea_4944_9125_fb2f7b67f95f.slice/crio-4dc4a2cadb600269004e62505d89b368a60c80d44f67c5295934b476cea920d0 WatchSource:0}: Error finding container 4dc4a2cadb600269004e62505d89b368a60c80d44f67c5295934b476cea920d0: Status 404 returned error can't find the container with id 4dc4a2cadb600269004e62505d89b368a60c80d44f67c5295934b476cea920d0
Sep 17 16:17:19.730941 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:19.730909    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"]
Sep 17 16:17:19.732608 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:17:19.732579    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a10d675_6269_4a2f_9665_4c83c666d18f.slice/crio-9bd0e09daa28bb44e6d15ef622c1432fc529fb64151964954190aa941a96ea09 WatchSource:0}: Error finding container 9bd0e09daa28bb44e6d15ef622c1432fc529fb64151964954190aa941a96ea09: Status 404 returned error can't find the container with id 9bd0e09daa28bb44e6d15ef622c1432fc529fb64151964954190aa941a96ea09
Sep 17 16:17:20.428951 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:20.428909    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2" event={"ID":"8a10d675-6269-4a2f-9665-4c83c666d18f","Type":"ContainerStarted","Data":"804551a1fb3b8f6944200c79d55b1f1ca8708b19b331d27798b9d176f40d0aaf"}
Sep 17 16:17:20.429415 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:20.428980    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2" event={"ID":"8a10d675-6269-4a2f-9665-4c83c666d18f","Type":"ContainerStarted","Data":"9bd0e09daa28bb44e6d15ef622c1432fc529fb64151964954190aa941a96ea09"}
Sep 17 16:17:20.429415 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:20.429021    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"
Sep 17 16:17:20.430751 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:20.430705    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" event={"ID":"acc1ca37-4dea-4944-9125-fb2f7b67f95f","Type":"ContainerStarted","Data":"8b0a37226620a16ccc4f820b3ea96876147b51a68f6828575ee24fcd81e6245f"}
Sep 17 16:17:20.430870 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:20.430767    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" event={"ID":"acc1ca37-4dea-4944-9125-fb2f7b67f95f","Type":"ContainerStarted","Data":"4dc4a2cadb600269004e62505d89b368a60c80d44f67c5295934b476cea920d0"}
Sep 17 16:17:20.448239 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:20.448185    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2" podStartSLOduration=1.448165054 podStartE2EDuration="1.448165054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:17:20.447085495 +0000 UTC m=+2305.918925353" watchObservedRunningTime="2026-09-17 16:17:20.448165054 +0000 UTC m=+2305.920004899"
Sep 17 16:17:23.454676 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:23.454635    2574 generic.go:356] "Generic (PLEG): container finished" podID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerID="8b0a37226620a16ccc4f820b3ea96876147b51a68f6828575ee24fcd81e6245f" exitCode=0
Sep 17 16:17:23.455046 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:23.454704    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" event={"ID":"acc1ca37-4dea-4944-9125-fb2f7b67f95f","Type":"ContainerDied","Data":"8b0a37226620a16ccc4f820b3ea96876147b51a68f6828575ee24fcd81e6245f"}
Sep 17 16:17:23.891902 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:23.891865    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:24.462241 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:24.462203    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" event={"ID":"acc1ca37-4dea-4944-9125-fb2f7b67f95f","Type":"ContainerStarted","Data":"3b81e8b3df2b4a1c369e961b50836f5f70640bd9dec191dfa3d1f6afbd9841d8"}
Sep 17 16:17:24.490122 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:24.490054    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" podStartSLOduration=3.528452982 podStartE2EDuration="6.490035416s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.961582434s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:17:24.485401501 +0000 UTC m=+2309.957241348" watchObservedRunningTime="2026-09-17 16:17:24.490035416 +0000 UTC m=+2309.961875262"
Sep 17 16:17:24.631786 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:24.631737    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:25.225252 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:25.225216    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:29.225398 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:29.225353    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:17:29.227180 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:29.227150    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 16:17:30.041636 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:30.041593    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:30.155752 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:30.155717    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:39.225315 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:39.225258    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 16:17:45.710312 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:45.710272    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:45.716526 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:45.716490    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"]
Sep 17 16:17:45.806888 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:45.806852    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:49.225627 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:49.225514    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 16:17:51.438301 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:51.438271    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"
Sep 17 16:17:52.750241 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:52.750199    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr" podUID="065e7b92-223c-4886-879e-0dfa0c6b5a84" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:17:59.226212 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:17:59.226164    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 16:18:00.740517 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:00.740467    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr" podUID="065e7b92-223c-4886-879e-0dfa0c6b5a84" containerName="main" containerID="cri-o://7ece17ff478a91dd9e85844a9abd3444f7c8b0936ea337a1a26188986092be8a" gracePeriod=45
Sep 17 16:18:02.750611 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:02.750568    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr" podUID="065e7b92-223c-4886-879e-0dfa0c6b5a84" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:18:09.225714 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:09.225659    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 16:18:12.750663 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:12.750617    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr" podUID="065e7b92-223c-4886-879e-0dfa0c6b5a84" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:18:12.751248 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:12.750743    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"
Sep 17 16:18:19.226238 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:19.226179    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 16:18:22.750791 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:22.750752    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr" podUID="065e7b92-223c-4886-879e-0dfa0c6b5a84" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:18:29.226208 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:29.226157    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 16:18:31.003000 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.002922    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"
Sep 17 16:18:31.159236 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.159182    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/065e7b92-223c-4886-879e-0dfa0c6b5a84-tls-certs\") pod \"065e7b92-223c-4886-879e-0dfa0c6b5a84\" (UID: \"065e7b92-223c-4886-879e-0dfa0c6b5a84\") "
Sep 17 16:18:31.159236 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.159225    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf5f2\" (UniqueName: \"kubernetes.io/projected/065e7b92-223c-4886-879e-0dfa0c6b5a84-kube-api-access-zf5f2\") pod \"065e7b92-223c-4886-879e-0dfa0c6b5a84\" (UID: \"065e7b92-223c-4886-879e-0dfa0c6b5a84\") "
Sep 17 16:18:31.161535 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.161504    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065e7b92-223c-4886-879e-0dfa0c6b5a84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "065e7b92-223c-4886-879e-0dfa0c6b5a84" (UID: "065e7b92-223c-4886-879e-0dfa0c6b5a84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:18:31.161671 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.161590    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065e7b92-223c-4886-879e-0dfa0c6b5a84-kube-api-access-zf5f2" (OuterVolumeSpecName: "kube-api-access-zf5f2") pod "065e7b92-223c-4886-879e-0dfa0c6b5a84" (UID: "065e7b92-223c-4886-879e-0dfa0c6b5a84"). InnerVolumeSpecName "kube-api-access-zf5f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:18:31.260587 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.260480    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/065e7b92-223c-4886-879e-0dfa0c6b5a84-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:18:31.260587 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.260515    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf5f2\" (UniqueName: \"kubernetes.io/projected/065e7b92-223c-4886-879e-0dfa0c6b5a84-kube-api-access-zf5f2\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:18:31.809101 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.809059    2574 generic.go:356] "Generic (PLEG): container finished" podID="065e7b92-223c-4886-879e-0dfa0c6b5a84" containerID="7ece17ff478a91dd9e85844a9abd3444f7c8b0936ea337a1a26188986092be8a" exitCode=0
Sep 17 16:18:31.809101 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.809106    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"
Sep 17 16:18:31.809385 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.809143    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr" event={"ID":"065e7b92-223c-4886-879e-0dfa0c6b5a84","Type":"ContainerDied","Data":"7ece17ff478a91dd9e85844a9abd3444f7c8b0936ea337a1a26188986092be8a"}
Sep 17 16:18:31.809385 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.809203    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr" event={"ID":"065e7b92-223c-4886-879e-0dfa0c6b5a84","Type":"ContainerDied","Data":"217711a0fd29bdad4fa3ec4eb1e581c13101ac2098f9ef60565e911bc4073f1f"}
Sep 17 16:18:31.809385 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.809229    2574 scope.go:117] "RemoveContainer" containerID="7ece17ff478a91dd9e85844a9abd3444f7c8b0936ea337a1a26188986092be8a"
Sep 17 16:18:31.820261 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.820241    2574 scope.go:117] "RemoveContainer" containerID="7ece17ff478a91dd9e85844a9abd3444f7c8b0936ea337a1a26188986092be8a"
Sep 17 16:18:31.820656 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:18:31.820631    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ece17ff478a91dd9e85844a9abd3444f7c8b0936ea337a1a26188986092be8a\": container with ID starting with 7ece17ff478a91dd9e85844a9abd3444f7c8b0936ea337a1a26188986092be8a not found: ID does not exist" containerID="7ece17ff478a91dd9e85844a9abd3444f7c8b0936ea337a1a26188986092be8a"
Sep 17 16:18:31.820751 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.820666    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ece17ff478a91dd9e85844a9abd3444f7c8b0936ea337a1a26188986092be8a"} err="failed to get container status \"7ece17ff478a91dd9e85844a9abd3444f7c8b0936ea337a1a26188986092be8a\": rpc error: code = NotFound desc = could not find container \"7ece17ff478a91dd9e85844a9abd3444f7c8b0936ea337a1a26188986092be8a\": container with ID starting with 7ece17ff478a91dd9e85844a9abd3444f7c8b0936ea337a1a26188986092be8a not found: ID does not exist"
Sep 17 16:18:31.850867 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.850823    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"]
Sep 17 16:18:31.857582 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:31.857548    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheg5dtr"]
Sep 17 16:18:33.048815 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:33.048775    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="065e7b92-223c-4886-879e-0dfa0c6b5a84" path="/var/lib/kubelet/pods/065e7b92-223c-4886-879e-0dfa0c6b5a84/volumes"
Sep 17 16:18:35.648479 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:35.648420    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:35.785158 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:35.785096    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:39.235244 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:39.235203    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:18:39.242914 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:39.242886    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:18:43.726108 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.726072    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"]
Sep 17 16:18:43.726580 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.726500    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065e7b92-223c-4886-879e-0dfa0c6b5a84" containerName="main"
Sep 17 16:18:43.726580 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.726512    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="065e7b92-223c-4886-879e-0dfa0c6b5a84" containerName="main"
Sep 17 16:18:43.726668 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.726600    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="065e7b92-223c-4886-879e-0dfa0c6b5a84" containerName="main"
Sep 17 16:18:43.731055 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.731034    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.737051 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.737027    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:18:43.738860 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.738836    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 17 16:18:43.738962 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.738891    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-9njkc\""
Sep 17 16:18:43.739396 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.739377    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 16:18:43.739525 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.739481    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 16:18:43.739602 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.739511    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 16:18:43.759689 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.759649    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"]
Sep 17 16:18:43.759905 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.759881    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:43.877491 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.877423    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-dshm\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.877491 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.877484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c193b381-0c6e-4cfe-b82e-8c351b10e126-tls-certs\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.877697 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.877521    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-home\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.877697 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.877536    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-tmp-dir\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.877697 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.877556    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.877697 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.877575    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c193b381-0c6e-4cfe-b82e-8c351b10e126-cabundle-cert\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.877697 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.877599    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlkq6\" (UniqueName: \"kubernetes.io/projected/c193b381-0c6e-4cfe-b82e-8c351b10e126-kube-api-access-wlkq6\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.877697 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.877618    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-model-cache\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.978766 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.978679    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-home\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.978766 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.978718    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-tmp-dir\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.978766 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.978736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.978766 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.978765    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c193b381-0c6e-4cfe-b82e-8c351b10e126-cabundle-cert\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.979090 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.978806    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlkq6\" (UniqueName: \"kubernetes.io/projected/c193b381-0c6e-4cfe-b82e-8c351b10e126-kube-api-access-wlkq6\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.979090 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.978835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-model-cache\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.979196 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.979124    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-dshm\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.979196 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.979151    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-home\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.979284 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.979224    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-tmp-dir\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.979284 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.979272    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.979379 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.979290    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c193b381-0c6e-4cfe-b82e-8c351b10e126-tls-certs\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.979379 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.979306    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-model-cache\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.979654 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.979545    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c193b381-0c6e-4cfe-b82e-8c351b10e126-cabundle-cert\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.981504 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.981481    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-dshm\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.981767 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.981750    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c193b381-0c6e-4cfe-b82e-8c351b10e126-tls-certs\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:43.988977 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:43.988955    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlkq6\" (UniqueName: \"kubernetes.io/projected/c193b381-0c6e-4cfe-b82e-8c351b10e126-kube-api-access-wlkq6\") pod \"custom-route-timeout-test-kserve-5f77c95c5b-7sqml\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:44.041831 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:44.041794    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:44.042952 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:44.042920    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:44.177950 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:44.177913    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:44.340179 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:44.340137    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"]
Sep 17 16:18:44.342886 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:18:44.342851    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc193b381_0c6e_4cfe_b82e_8c351b10e126.slice/crio-e0ba27d04874281d210793b16512f61640cfa1011b0246e99a46407ac735649b WatchSource:0}: Error finding container e0ba27d04874281d210793b16512f61640cfa1011b0246e99a46407ac735649b: Status 404 returned error can't find the container with id e0ba27d04874281d210793b16512f61640cfa1011b0246e99a46407ac735649b
Sep 17 16:18:44.879861 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:44.879827    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" event={"ID":"c193b381-0c6e-4cfe-b82e-8c351b10e126","Type":"ContainerStarted","Data":"d4f76d5e19c7f84b0e71f1fd4d58cc161a1d67f1fbf71c20bddd3278572affeb"}
Sep 17 16:18:44.880241 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:44.879880    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" event={"ID":"c193b381-0c6e-4cfe-b82e-8c351b10e126","Type":"ContainerStarted","Data":"e0ba27d04874281d210793b16512f61640cfa1011b0246e99a46407ac735649b"}
Sep 17 16:18:47.897552 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:47.897515    2574 generic.go:356] "Generic (PLEG): container finished" podID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerID="d4f76d5e19c7f84b0e71f1fd4d58cc161a1d67f1fbf71c20bddd3278572affeb" exitCode=0
Sep 17 16:18:47.897921 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:47.897589    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" event={"ID":"c193b381-0c6e-4cfe-b82e-8c351b10e126","Type":"ContainerDied","Data":"d4f76d5e19c7f84b0e71f1fd4d58cc161a1d67f1fbf71c20bddd3278572affeb"}
Sep 17 16:18:48.904079 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:48.904040    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" event={"ID":"c193b381-0c6e-4cfe-b82e-8c351b10e126","Type":"ContainerStarted","Data":"cebd6d5389c7792747a33f35df9ea6c5a3db6bad8878a1752da0e10e4b891580"}
Sep 17 16:18:49.042179 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:49.042140    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:49.868321 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:49.868285    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:51.635594 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:51.635559    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:51.743801 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:51.743746    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:54.042790 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:54.042746    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:18:54.044574 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:18:54.044543    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 16:19:04.043192 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:04.043143    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 16:19:09.605796 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:09.605751    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:09.671427 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:09.671385    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:14.043289 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:14.043186    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 16:19:18.781535 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:18.781416    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:18.791705 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:18.791641    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" podStartSLOduration=33.313096383 podStartE2EDuration="35.791617367s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.478520984s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:18:48.951076854 +0000 UTC m=+2394.422916700" watchObservedRunningTime="2026-09-17 16:19:18.791617367 +0000 UTC m=+2424.263457215"
Sep 17 16:19:18.792525 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:18.792504    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"]
Sep 17 16:19:18.815531 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:18.815495    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"]
Sep 17 16:19:18.870462 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:18.870406    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:21.438561 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:21.438524    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2" podUID="8a10d675-6269-4a2f-9665-4c83c666d18f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:19:24.042918 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:24.042872    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 16:19:31.438498 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:31.438433    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2" podUID="8a10d675-6269-4a2f-9665-4c83c666d18f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:19:33.816865 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:33.816801    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerName="main" containerID="cri-o://3b81e8b3df2b4a1c369e961b50836f5f70640bd9dec191dfa3d1f6afbd9841d8" gracePeriod=45
Sep 17 16:19:33.834931 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:33.834887    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2" podUID="8a10d675-6269-4a2f-9665-4c83c666d18f" containerName="main" containerID="cri-o://804551a1fb3b8f6944200c79d55b1f1ca8708b19b331d27798b9d176f40d0aaf" gracePeriod=45
Sep 17 16:19:34.042948 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.042892    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 16:19:34.236844 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.236793    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 17 16:19:34.476690 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.476665    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:19:34.544497 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.544385    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-home\") pod \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") "
Sep 17 16:19:34.544497 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.544483    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acc1ca37-4dea-4944-9125-fb2f7b67f95f-cabundle-cert\") pod \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") "
Sep 17 16:19:34.544721 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.544532    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b5n7\" (UniqueName: \"kubernetes.io/projected/acc1ca37-4dea-4944-9125-fb2f7b67f95f-kube-api-access-2b5n7\") pod \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") "
Sep 17 16:19:34.544721 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.544595    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-dshm\") pod \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") "
Sep 17 16:19:34.544721 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.544628    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-tmp-dir\") pod \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") "
Sep 17 16:19:34.544721 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.544696    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-kserve-provision-location\") pod \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") "
Sep 17 16:19:34.544942 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.544730    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acc1ca37-4dea-4944-9125-fb2f7b67f95f-tls-certs\") pod \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") "
Sep 17 16:19:34.544942 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.544760    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-model-cache\") pod \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\" (UID: \"acc1ca37-4dea-4944-9125-fb2f7b67f95f\") "
Sep 17 16:19:34.544942 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.544818    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-home" (OuterVolumeSpecName: "home") pod "acc1ca37-4dea-4944-9125-fb2f7b67f95f" (UID: "acc1ca37-4dea-4944-9125-fb2f7b67f95f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:19:34.545095 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.545040    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-model-cache" (OuterVolumeSpecName: "model-cache") pod "acc1ca37-4dea-4944-9125-fb2f7b67f95f" (UID: "acc1ca37-4dea-4944-9125-fb2f7b67f95f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:19:34.545156 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.545093    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:19:34.545283 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.545250    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc1ca37-4dea-4944-9125-fb2f7b67f95f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "acc1ca37-4dea-4944-9125-fb2f7b67f95f" (UID: "acc1ca37-4dea-4944-9125-fb2f7b67f95f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:19:34.547141 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.547113    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc1ca37-4dea-4944-9125-fb2f7b67f95f-kube-api-access-2b5n7" (OuterVolumeSpecName: "kube-api-access-2b5n7") pod "acc1ca37-4dea-4944-9125-fb2f7b67f95f" (UID: "acc1ca37-4dea-4944-9125-fb2f7b67f95f"). InnerVolumeSpecName "kube-api-access-2b5n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:19:34.547262 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.547149    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc1ca37-4dea-4944-9125-fb2f7b67f95f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "acc1ca37-4dea-4944-9125-fb2f7b67f95f" (UID: "acc1ca37-4dea-4944-9125-fb2f7b67f95f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:19:34.547327 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.547306    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-dshm" (OuterVolumeSpecName: "dshm") pod "acc1ca37-4dea-4944-9125-fb2f7b67f95f" (UID: "acc1ca37-4dea-4944-9125-fb2f7b67f95f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:19:34.563943 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.563905    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "acc1ca37-4dea-4944-9125-fb2f7b67f95f" (UID: "acc1ca37-4dea-4944-9125-fb2f7b67f95f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:19:34.574576 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.574535    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "acc1ca37-4dea-4944-9125-fb2f7b67f95f" (UID: "acc1ca37-4dea-4944-9125-fb2f7b67f95f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:19:34.645919 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.645886    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:19:34.645919 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.645913    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:19:34.645919 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.645924    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:19:34.646158 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.645936    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acc1ca37-4dea-4944-9125-fb2f7b67f95f-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:19:34.646158 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.645945    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acc1ca37-4dea-4944-9125-fb2f7b67f95f-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:19:34.646158 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.645952    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acc1ca37-4dea-4944-9125-fb2f7b67f95f-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:19:34.646158 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:34.645961    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b5n7\" (UniqueName: \"kubernetes.io/projected/acc1ca37-4dea-4944-9125-fb2f7b67f95f-kube-api-access-2b5n7\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:19:35.168273 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:35.168233    2574 generic.go:356] "Generic (PLEG): container finished" podID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerID="3b81e8b3df2b4a1c369e961b50836f5f70640bd9dec191dfa3d1f6afbd9841d8" exitCode=0
Sep 17 16:19:35.168773 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:35.168291    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" event={"ID":"acc1ca37-4dea-4944-9125-fb2f7b67f95f","Type":"ContainerDied","Data":"3b81e8b3df2b4a1c369e961b50836f5f70640bd9dec191dfa3d1f6afbd9841d8"}
Sep 17 16:19:35.168773 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:35.168303    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"
Sep 17 16:19:35.168773 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:35.168356    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx" event={"ID":"acc1ca37-4dea-4944-9125-fb2f7b67f95f","Type":"ContainerDied","Data":"4dc4a2cadb600269004e62505d89b368a60c80d44f67c5295934b476cea920d0"}
Sep 17 16:19:35.168773 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:35.168378    2574 scope.go:117] "RemoveContainer" containerID="3b81e8b3df2b4a1c369e961b50836f5f70640bd9dec191dfa3d1f6afbd9841d8"
Sep 17 16:19:35.178929 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:35.178906    2574 scope.go:117] "RemoveContainer" containerID="8b0a37226620a16ccc4f820b3ea96876147b51a68f6828575ee24fcd81e6245f"
Sep 17 16:19:35.188065 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:35.188037    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"]
Sep 17 16:19:35.188255 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:35.188240    2574 scope.go:117] "RemoveContainer" containerID="3b81e8b3df2b4a1c369e961b50836f5f70640bd9dec191dfa3d1f6afbd9841d8"
Sep 17 16:19:35.188591 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:19:35.188572    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b81e8b3df2b4a1c369e961b50836f5f70640bd9dec191dfa3d1f6afbd9841d8\": container with ID starting with 3b81e8b3df2b4a1c369e961b50836f5f70640bd9dec191dfa3d1f6afbd9841d8 not found: ID does not exist" containerID="3b81e8b3df2b4a1c369e961b50836f5f70640bd9dec191dfa3d1f6afbd9841d8"
Sep 17 16:19:35.188649 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:35.188599    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b81e8b3df2b4a1c369e961b50836f5f70640bd9dec191dfa3d1f6afbd9841d8"} err="failed to get container status \"3b81e8b3df2b4a1c369e961b50836f5f70640bd9dec191dfa3d1f6afbd9841d8\": rpc error: code = NotFound desc = could not find container \"3b81e8b3df2b4a1c369e961b50836f5f70640bd9dec191dfa3d1f6afbd9841d8\": container with ID starting with 3b81e8b3df2b4a1c369e961b50836f5f70640bd9dec191dfa3d1f6afbd9841d8 not found: ID does not exist"
Sep 17 16:19:35.188649 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:35.188620    2574 scope.go:117] "RemoveContainer" containerID="8b0a37226620a16ccc4f820b3ea96876147b51a68f6828575ee24fcd81e6245f"
Sep 17 16:19:35.188901 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:19:35.188883    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0a37226620a16ccc4f820b3ea96876147b51a68f6828575ee24fcd81e6245f\": container with ID starting with 8b0a37226620a16ccc4f820b3ea96876147b51a68f6828575ee24fcd81e6245f not found: ID does not exist" containerID="8b0a37226620a16ccc4f820b3ea96876147b51a68f6828575ee24fcd81e6245f"
Sep 17 16:19:35.188966 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:35.188904    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0a37226620a16ccc4f820b3ea96876147b51a68f6828575ee24fcd81e6245f"} err="failed to get container status \"8b0a37226620a16ccc4f820b3ea96876147b51a68f6828575ee24fcd81e6245f\": rpc error: code = NotFound desc = could not find container \"8b0a37226620a16ccc4f820b3ea96876147b51a68f6828575ee24fcd81e6245f\": container with ID starting with 8b0a37226620a16ccc4f820b3ea96876147b51a68f6828575ee24fcd81e6245f not found: ID does not exist"
Sep 17 16:19:35.191966 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:35.191943    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-644f6dc46c-lqvqx"]
Sep 17 16:19:36.950471 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:36.950408    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:37.012481 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:37.012420    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:37.050000 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:37.049967    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" path="/var/lib/kubelet/pods/acc1ca37-4dea-4944-9125-fb2f7b67f95f/volumes"
Sep 17 16:19:41.438673 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:41.438637    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2" podUID="8a10d675-6269-4a2f-9665-4c83c666d18f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:19:41.439081 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:41.438736    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"
Sep 17 16:19:44.042623 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:44.042583    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 16:19:51.439136 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:51.439091    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2" podUID="8a10d675-6269-4a2f-9665-4c83c666d18f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:19:54.042330 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:19:54.042288    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 16:20:01.439205 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:01.439158    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2" podUID="8a10d675-6269-4a2f-9665-4c83c666d18f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:20:04.042421 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.042379    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 16:20:04.096346 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.096278    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"
Sep 17 16:20:04.229299 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.229264    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a10d675-6269-4a2f-9665-4c83c666d18f-tls-certs\") pod \"8a10d675-6269-4a2f-9665-4c83c666d18f\" (UID: \"8a10d675-6269-4a2f-9665-4c83c666d18f\") "
Sep 17 16:20:04.229531 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.229318    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5hp9\" (UniqueName: \"kubernetes.io/projected/8a10d675-6269-4a2f-9665-4c83c666d18f-kube-api-access-p5hp9\") pod \"8a10d675-6269-4a2f-9665-4c83c666d18f\" (UID: \"8a10d675-6269-4a2f-9665-4c83c666d18f\") "
Sep 17 16:20:04.231598 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.231558    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a10d675-6269-4a2f-9665-4c83c666d18f-kube-api-access-p5hp9" (OuterVolumeSpecName: "kube-api-access-p5hp9") pod "8a10d675-6269-4a2f-9665-4c83c666d18f" (UID: "8a10d675-6269-4a2f-9665-4c83c666d18f"). InnerVolumeSpecName "kube-api-access-p5hp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:20:04.231732 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.231628    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a10d675-6269-4a2f-9665-4c83c666d18f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a10d675-6269-4a2f-9665-4c83c666d18f" (UID: "8a10d675-6269-4a2f-9665-4c83c666d18f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:20:04.325423 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.325390    2574 generic.go:356] "Generic (PLEG): container finished" podID="8a10d675-6269-4a2f-9665-4c83c666d18f" containerID="804551a1fb3b8f6944200c79d55b1f1ca8708b19b331d27798b9d176f40d0aaf" exitCode=0
Sep 17 16:20:04.325648 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.325430    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"
Sep 17 16:20:04.325648 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.325483    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2" event={"ID":"8a10d675-6269-4a2f-9665-4c83c666d18f","Type":"ContainerDied","Data":"804551a1fb3b8f6944200c79d55b1f1ca8708b19b331d27798b9d176f40d0aaf"}
Sep 17 16:20:04.325648 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.325545    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2" event={"ID":"8a10d675-6269-4a2f-9665-4c83c666d18f","Type":"ContainerDied","Data":"9bd0e09daa28bb44e6d15ef622c1432fc529fb64151964954190aa941a96ea09"}
Sep 17 16:20:04.325648 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.325574    2574 scope.go:117] "RemoveContainer" containerID="804551a1fb3b8f6944200c79d55b1f1ca8708b19b331d27798b9d176f40d0aaf"
Sep 17 16:20:04.330097 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.330075    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a10d675-6269-4a2f-9665-4c83c666d18f-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:20:04.330201 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.330099    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5hp9\" (UniqueName: \"kubernetes.io/projected/8a10d675-6269-4a2f-9665-4c83c666d18f-kube-api-access-p5hp9\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:20:04.335052 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.335032    2574 scope.go:117] "RemoveContainer" containerID="804551a1fb3b8f6944200c79d55b1f1ca8708b19b331d27798b9d176f40d0aaf"
Sep 17 16:20:04.335302 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:20:04.335286    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804551a1fb3b8f6944200c79d55b1f1ca8708b19b331d27798b9d176f40d0aaf\": container with ID starting with 804551a1fb3b8f6944200c79d55b1f1ca8708b19b331d27798b9d176f40d0aaf not found: ID does not exist" containerID="804551a1fb3b8f6944200c79d55b1f1ca8708b19b331d27798b9d176f40d0aaf"
Sep 17 16:20:04.335353 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.335308    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804551a1fb3b8f6944200c79d55b1f1ca8708b19b331d27798b9d176f40d0aaf"} err="failed to get container status \"804551a1fb3b8f6944200c79d55b1f1ca8708b19b331d27798b9d176f40d0aaf\": rpc error: code = NotFound desc = could not find container \"804551a1fb3b8f6944200c79d55b1f1ca8708b19b331d27798b9d176f40d0aaf\": container with ID starting with 804551a1fb3b8f6944200c79d55b1f1ca8708b19b331d27798b9d176f40d0aaf not found: ID does not exist"
Sep 17 16:20:04.349372 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.349298    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"]
Sep 17 16:20:04.355209 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:04.355179    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepzlg2"]
Sep 17 16:20:05.048544 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:05.048508    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a10d675-6269-4a2f-9665-4c83c666d18f" path="/var/lib/kubelet/pods/8a10d675-6269-4a2f-9665-4c83c666d18f/volumes"
Sep 17 16:20:08.037289 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:08.037250    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:08.370655 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:08.370616    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:09.011013 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:09.010977    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:11.803590 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.803552    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"]
Sep 17 16:20:11.804061 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.804036    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a10d675-6269-4a2f-9665-4c83c666d18f" containerName="main"
Sep 17 16:20:11.804191 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.804078    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a10d675-6269-4a2f-9665-4c83c666d18f" containerName="main"
Sep 17 16:20:11.804191 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.804119    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerName="main"
Sep 17 16:20:11.804191 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.804129    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerName="main"
Sep 17 16:20:11.804191 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.804146    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerName="storage-initializer"
Sep 17 16:20:11.804191 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.804155    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerName="storage-initializer"
Sep 17 16:20:11.804434 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.804302    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a10d675-6269-4a2f-9665-4c83c666d18f" containerName="main"
Sep 17 16:20:11.804434 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.804317    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="acc1ca37-4dea-4944-9125-fb2f7b67f95f" containerName="main"
Sep 17 16:20:11.807682 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.807667    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:11.809929 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.809908    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 17 16:20:11.810764 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.810738    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 16:20:11.810932 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.810912    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 16:20:11.811025 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.811008    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-2vsbs\""
Sep 17 16:20:11.811128 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.811111    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 16:20:11.811396 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.811379    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:20:11.823602 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.823575    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"]
Sep 17 16:20:11.868977 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.868943    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:11.900415 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.900374    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk2ws\" (UniqueName: \"kubernetes.io/projected/edc71fd8-61bc-43ad-9e72-591c0959860b-kube-api-access-wk2ws\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:11.900601 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.900422    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:11.900601 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.900476    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:11.900601 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.900501    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:11.900601 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.900590    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:11.900774 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.900617    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:11.900774 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.900636    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edc71fd8-61bc-43ad-9e72-591c0959860b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:11.900774 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:11.900661    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc71fd8-61bc-43ad-9e72-591c0959860b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.001754 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.001712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk2ws\" (UniqueName: \"kubernetes.io/projected/edc71fd8-61bc-43ad-9e72-591c0959860b-kube-api-access-wk2ws\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.001754 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.001759    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.002002 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.001777    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.002002 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.001796    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.002002 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.001839    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.002002 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.001856    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.002002 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.001873    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edc71fd8-61bc-43ad-9e72-591c0959860b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.002002 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.001893    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc71fd8-61bc-43ad-9e72-591c0959860b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.002353 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.002255    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.002353 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.002273    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.002495 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.002351    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.002646 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.002623    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.002803 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.002779    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edc71fd8-61bc-43ad-9e72-591c0959860b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.004496 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.004479    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.004564 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.004548    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc71fd8-61bc-43ad-9e72-591c0959860b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.010475 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.010427    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk2ws\" (UniqueName: \"kubernetes.io/projected/edc71fd8-61bc-43ad-9e72-591c0959860b-kube-api-access-wk2ws\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.070956 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.070859    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:12.121932 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.121890    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:12.408656 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:12.408623    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"]
Sep 17 16:20:12.411305 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:20:12.411277    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc71fd8_61bc_43ad_9e72_591c0959860b.slice/crio-a653d1c9a1454f0c6e6fcdf7dd809c2fe741b105ac1c7a1d4ee776beef04c98d WatchSource:0}: Error finding container a653d1c9a1454f0c6e6fcdf7dd809c2fe741b105ac1c7a1d4ee776beef04c98d: Status 404 returned error can't find the container with id a653d1c9a1454f0c6e6fcdf7dd809c2fe741b105ac1c7a1d4ee776beef04c98d
Sep 17 16:20:13.376312 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:13.376271    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" event={"ID":"edc71fd8-61bc-43ad-9e72-591c0959860b","Type":"ContainerStarted","Data":"91d14e35773b979462d5dacf7bb05aa932832b7e8d149501ecc8d62fd1428393"}
Sep 17 16:20:13.376809 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:13.376329    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" event={"ID":"edc71fd8-61bc-43ad-9e72-591c0959860b","Type":"ContainerStarted","Data":"a653d1c9a1454f0c6e6fcdf7dd809c2fe741b105ac1c7a1d4ee776beef04c98d"}
Sep 17 16:20:14.043231 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:14.043142    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 16:20:15.398417 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:15.398378    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:16.393149 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:16.393117    2574 generic.go:356] "Generic (PLEG): container finished" podID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerID="91d14e35773b979462d5dacf7bb05aa932832b7e8d149501ecc8d62fd1428393" exitCode=0
Sep 17 16:20:16.393330 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:16.393197    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" event={"ID":"edc71fd8-61bc-43ad-9e72-591c0959860b","Type":"ContainerDied","Data":"91d14e35773b979462d5dacf7bb05aa932832b7e8d149501ecc8d62fd1428393"}
Sep 17 16:20:17.400322 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:17.400282    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" event={"ID":"edc71fd8-61bc-43ad-9e72-591c0959860b","Type":"ContainerStarted","Data":"04a11eb05d1961dc895367323fc7245c49007abf37cfd8a82b5e1c06214fc6fd"}
Sep 17 16:20:17.425633 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:17.425566    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" podStartSLOduration=3.421001609 podStartE2EDuration="6.425545269s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.00454366s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:20:17.421842716 +0000 UTC m=+2482.893682587" watchObservedRunningTime="2026-09-17 16:20:17.425545269 +0000 UTC m=+2482.897385116"
Sep 17 16:20:18.122054 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:18.122019    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:21.406729 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:21.406692    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:21.479116 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:21.479079    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:22.122732 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:22.122689    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:20:22.124661 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:22.124621    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 16:20:24.043381 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:24.043333    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 16:20:32.122359 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:32.122307    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 16:20:33.632331 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:33.632293    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:34.042978 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:34.042880    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 16:20:42.122666 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:42.122593    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 16:20:44.053867 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:44.053830    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:20:44.062279 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:44.062245    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:20:50.460804 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:50.460719    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"]
Sep 17 16:20:50.465060 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:50.464979    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:50.556822 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:50.556781    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:52.122764 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:52.122720    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 16:20:53.599963 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:20:53.599917    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:02.122569 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:02.122513    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 16:21:05.490400 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:05.490323    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main" containerID="cri-o://cebd6d5389c7792747a33f35df9ea6c5a3db6bad8878a1752da0e10e4b891580" gracePeriod=45
Sep 17 16:21:06.054967 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.054923    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 16:21:06.249520 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.249456    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:21:06.308575 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.308538    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-home\") pod \"c193b381-0c6e-4cfe-b82e-8c351b10e126\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") "
Sep 17 16:21:06.308762 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.308586    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c193b381-0c6e-4cfe-b82e-8c351b10e126-cabundle-cert\") pod \"c193b381-0c6e-4cfe-b82e-8c351b10e126\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") "
Sep 17 16:21:06.308762 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.308627    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-kserve-provision-location\") pod \"c193b381-0c6e-4cfe-b82e-8c351b10e126\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") "
Sep 17 16:21:06.308762 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.308718    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c193b381-0c6e-4cfe-b82e-8c351b10e126-tls-certs\") pod \"c193b381-0c6e-4cfe-b82e-8c351b10e126\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") "
Sep 17 16:21:06.308762 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.308753    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-model-cache\") pod \"c193b381-0c6e-4cfe-b82e-8c351b10e126\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") "
Sep 17 16:21:06.308983 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.308803    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-dshm\") pod \"c193b381-0c6e-4cfe-b82e-8c351b10e126\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") "
Sep 17 16:21:06.308983 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.308867    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-tmp-dir\") pod \"c193b381-0c6e-4cfe-b82e-8c351b10e126\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") "
Sep 17 16:21:06.308983 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.308899    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlkq6\" (UniqueName: \"kubernetes.io/projected/c193b381-0c6e-4cfe-b82e-8c351b10e126-kube-api-access-wlkq6\") pod \"c193b381-0c6e-4cfe-b82e-8c351b10e126\" (UID: \"c193b381-0c6e-4cfe-b82e-8c351b10e126\") "
Sep 17 16:21:06.309139 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.309003    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-model-cache" (OuterVolumeSpecName: "model-cache") pod "c193b381-0c6e-4cfe-b82e-8c351b10e126" (UID: "c193b381-0c6e-4cfe-b82e-8c351b10e126"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:21:06.309200 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.309008    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c193b381-0c6e-4cfe-b82e-8c351b10e126-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c193b381-0c6e-4cfe-b82e-8c351b10e126" (UID: "c193b381-0c6e-4cfe-b82e-8c351b10e126"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:21:06.309258 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.309237    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:06.309758 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.309730    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-home" (OuterVolumeSpecName: "home") pod "c193b381-0c6e-4cfe-b82e-8c351b10e126" (UID: "c193b381-0c6e-4cfe-b82e-8c351b10e126"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:21:06.311350 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.311316    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-dshm" (OuterVolumeSpecName: "dshm") pod "c193b381-0c6e-4cfe-b82e-8c351b10e126" (UID: "c193b381-0c6e-4cfe-b82e-8c351b10e126"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:21:06.311528 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.311504    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c193b381-0c6e-4cfe-b82e-8c351b10e126-kube-api-access-wlkq6" (OuterVolumeSpecName: "kube-api-access-wlkq6") pod "c193b381-0c6e-4cfe-b82e-8c351b10e126" (UID: "c193b381-0c6e-4cfe-b82e-8c351b10e126"). InnerVolumeSpecName "kube-api-access-wlkq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:21:06.311601 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.311535    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c193b381-0c6e-4cfe-b82e-8c351b10e126-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c193b381-0c6e-4cfe-b82e-8c351b10e126" (UID: "c193b381-0c6e-4cfe-b82e-8c351b10e126"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:21:06.322372 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.322338    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c193b381-0c6e-4cfe-b82e-8c351b10e126" (UID: "c193b381-0c6e-4cfe-b82e-8c351b10e126"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:21:06.340567 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.340529    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c193b381-0c6e-4cfe-b82e-8c351b10e126" (UID: "c193b381-0c6e-4cfe-b82e-8c351b10e126"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:21:06.409821 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.409786    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c193b381-0c6e-4cfe-b82e-8c351b10e126-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:06.409821 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.409814    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:06.409821 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.409826    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:06.410039 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.409834    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlkq6\" (UniqueName: \"kubernetes.io/projected/c193b381-0c6e-4cfe-b82e-8c351b10e126-kube-api-access-wlkq6\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:06.410039 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.409843    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:06.410039 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.409852    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c193b381-0c6e-4cfe-b82e-8c351b10e126-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:06.410039 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.409860    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c193b381-0c6e-4cfe-b82e-8c351b10e126-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:06.679247 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.679206    2574 generic.go:356] "Generic (PLEG): container finished" podID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerID="cebd6d5389c7792747a33f35df9ea6c5a3db6bad8878a1752da0e10e4b891580" exitCode=0
Sep 17 16:21:06.679730 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.679256    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"
Sep 17 16:21:06.679730 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.679287    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" event={"ID":"c193b381-0c6e-4cfe-b82e-8c351b10e126","Type":"ContainerDied","Data":"cebd6d5389c7792747a33f35df9ea6c5a3db6bad8878a1752da0e10e4b891580"}
Sep 17 16:21:06.679730 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.679355    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml" event={"ID":"c193b381-0c6e-4cfe-b82e-8c351b10e126","Type":"ContainerDied","Data":"e0ba27d04874281d210793b16512f61640cfa1011b0246e99a46407ac735649b"}
Sep 17 16:21:06.679730 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.679393    2574 scope.go:117] "RemoveContainer" containerID="cebd6d5389c7792747a33f35df9ea6c5a3db6bad8878a1752da0e10e4b891580"
Sep 17 16:21:06.689006 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.688978    2574 scope.go:117] "RemoveContainer" containerID="d4f76d5e19c7f84b0e71f1fd4d58cc161a1d67f1fbf71c20bddd3278572affeb"
Sep 17 16:21:06.697674 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.697652    2574 scope.go:117] "RemoveContainer" containerID="cebd6d5389c7792747a33f35df9ea6c5a3db6bad8878a1752da0e10e4b891580"
Sep 17 16:21:06.697972 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:21:06.697945    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cebd6d5389c7792747a33f35df9ea6c5a3db6bad8878a1752da0e10e4b891580\": container with ID starting with cebd6d5389c7792747a33f35df9ea6c5a3db6bad8878a1752da0e10e4b891580 not found: ID does not exist" containerID="cebd6d5389c7792747a33f35df9ea6c5a3db6bad8878a1752da0e10e4b891580"
Sep 17 16:21:06.698073 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.697981    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cebd6d5389c7792747a33f35df9ea6c5a3db6bad8878a1752da0e10e4b891580"} err="failed to get container status \"cebd6d5389c7792747a33f35df9ea6c5a3db6bad8878a1752da0e10e4b891580\": rpc error: code = NotFound desc = could not find container \"cebd6d5389c7792747a33f35df9ea6c5a3db6bad8878a1752da0e10e4b891580\": container with ID starting with cebd6d5389c7792747a33f35df9ea6c5a3db6bad8878a1752da0e10e4b891580 not found: ID does not exist"
Sep 17 16:21:06.698073 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.698008    2574 scope.go:117] "RemoveContainer" containerID="d4f76d5e19c7f84b0e71f1fd4d58cc161a1d67f1fbf71c20bddd3278572affeb"
Sep 17 16:21:06.698307 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:21:06.698288    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f76d5e19c7f84b0e71f1fd4d58cc161a1d67f1fbf71c20bddd3278572affeb\": container with ID starting with d4f76d5e19c7f84b0e71f1fd4d58cc161a1d67f1fbf71c20bddd3278572affeb not found: ID does not exist" containerID="d4f76d5e19c7f84b0e71f1fd4d58cc161a1d67f1fbf71c20bddd3278572affeb"
Sep 17 16:21:06.698352 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.698313    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f76d5e19c7f84b0e71f1fd4d58cc161a1d67f1fbf71c20bddd3278572affeb"} err="failed to get container status \"d4f76d5e19c7f84b0e71f1fd4d58cc161a1d67f1fbf71c20bddd3278572affeb\": rpc error: code = NotFound desc = could not find container \"d4f76d5e19c7f84b0e71f1fd4d58cc161a1d67f1fbf71c20bddd3278572affeb\": container with ID starting with d4f76d5e19c7f84b0e71f1fd4d58cc161a1d67f1fbf71c20bddd3278572affeb not found: ID does not exist"
Sep 17 16:21:06.704347 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.704320    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"]
Sep 17 16:21:06.710192 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:06.710156    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5f77c95c5b-7sqml"]
Sep 17 16:21:07.048994 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:07.048916    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" path="/var/lib/kubelet/pods/c193b381-0c6e-4cfe-b82e-8c351b10e126/volumes"
Sep 17 16:21:12.122490 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:12.122421    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 16:21:14.242515 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:14.242471    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:15.472361 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:15.472325    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:16.663313 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:16.663273    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:16.731807 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:16.731772    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:22.123284 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:22.123236    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 16:21:32.132845 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:32.132807    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:21:32.140453 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:32.140415    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:21:33.815577 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:33.815539    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"]
Sep 17 16:21:33.828181 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:33.828143    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:35.764693 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:35.764658    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:40.370809 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:40.370769    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:40.471952 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:40.471913    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:40.638254 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:40.638175    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:40.737805 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:40.737772    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:44.096076 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:44.096038    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:47.923100 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:47.923061    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:48.849469 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:48.849413    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerName="main" containerID="cri-o://04a11eb05d1961dc895367323fc7245c49007abf37cfd8a82b5e1c06214fc6fd" gracePeriod=45
Sep 17 16:21:49.133804 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.133761    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 16:21:49.279136 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:21:49.279099    2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc71fd8_61bc_43ad_9e72_591c0959860b.slice/crio-04a11eb05d1961dc895367323fc7245c49007abf37cfd8a82b5e1c06214fc6fd.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 16:21:49.423819 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.423793    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:21:49.493115 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.493081    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-dshm\") pod \"edc71fd8-61bc-43ad-9e72-591c0959860b\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") "
Sep 17 16:21:49.493115 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.493115    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-kserve-provision-location\") pod \"edc71fd8-61bc-43ad-9e72-591c0959860b\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") "
Sep 17 16:21:49.493365 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.493135    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc71fd8-61bc-43ad-9e72-591c0959860b-tls-certs\") pod \"edc71fd8-61bc-43ad-9e72-591c0959860b\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") "
Sep 17 16:21:49.493365 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.493168    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edc71fd8-61bc-43ad-9e72-591c0959860b-cabundle-cert\") pod \"edc71fd8-61bc-43ad-9e72-591c0959860b\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") "
Sep 17 16:21:49.493365 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.493186    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-tmp-dir\") pod \"edc71fd8-61bc-43ad-9e72-591c0959860b\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") "
Sep 17 16:21:49.493365 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.493214    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-model-cache\") pod \"edc71fd8-61bc-43ad-9e72-591c0959860b\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") "
Sep 17 16:21:49.493365 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.493235    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-home\") pod \"edc71fd8-61bc-43ad-9e72-591c0959860b\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") "
Sep 17 16:21:49.493365 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.493307    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk2ws\" (UniqueName: \"kubernetes.io/projected/edc71fd8-61bc-43ad-9e72-591c0959860b-kube-api-access-wk2ws\") pod \"edc71fd8-61bc-43ad-9e72-591c0959860b\" (UID: \"edc71fd8-61bc-43ad-9e72-591c0959860b\") "
Sep 17 16:21:49.493684 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.493605    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc71fd8-61bc-43ad-9e72-591c0959860b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "edc71fd8-61bc-43ad-9e72-591c0959860b" (UID: "edc71fd8-61bc-43ad-9e72-591c0959860b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:21:49.494056 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.493999    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-model-cache" (OuterVolumeSpecName: "model-cache") pod "edc71fd8-61bc-43ad-9e72-591c0959860b" (UID: "edc71fd8-61bc-43ad-9e72-591c0959860b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:21:49.494056 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.494037    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "edc71fd8-61bc-43ad-9e72-591c0959860b" (UID: "edc71fd8-61bc-43ad-9e72-591c0959860b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:21:49.494215 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.494094    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-home" (OuterVolumeSpecName: "home") pod "edc71fd8-61bc-43ad-9e72-591c0959860b" (UID: "edc71fd8-61bc-43ad-9e72-591c0959860b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:21:49.495710 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.495682    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc71fd8-61bc-43ad-9e72-591c0959860b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edc71fd8-61bc-43ad-9e72-591c0959860b" (UID: "edc71fd8-61bc-43ad-9e72-591c0959860b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:21:49.495957 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.495936    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-dshm" (OuterVolumeSpecName: "dshm") pod "edc71fd8-61bc-43ad-9e72-591c0959860b" (UID: "edc71fd8-61bc-43ad-9e72-591c0959860b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:21:49.496030 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.496010    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc71fd8-61bc-43ad-9e72-591c0959860b-kube-api-access-wk2ws" (OuterVolumeSpecName: "kube-api-access-wk2ws") pod "edc71fd8-61bc-43ad-9e72-591c0959860b" (UID: "edc71fd8-61bc-43ad-9e72-591c0959860b"). InnerVolumeSpecName "kube-api-access-wk2ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:21:49.518293 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.518258    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "edc71fd8-61bc-43ad-9e72-591c0959860b" (UID: "edc71fd8-61bc-43ad-9e72-591c0959860b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:21:49.594988 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.594941    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:49.594988 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.594977    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:49.594988 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.594990    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk2ws\" (UniqueName: \"kubernetes.io/projected/edc71fd8-61bc-43ad-9e72-591c0959860b-kube-api-access-wk2ws\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:49.595245 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.595003    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:49.595245 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.595016    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:49.595245 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.595030    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc71fd8-61bc-43ad-9e72-591c0959860b-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:49.595245 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.595042    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edc71fd8-61bc-43ad-9e72-591c0959860b-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:49.595245 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.595052    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc71fd8-61bc-43ad-9e72-591c0959860b-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:21:49.903760 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.903722    2574 generic.go:356] "Generic (PLEG): container finished" podID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerID="04a11eb05d1961dc895367323fc7245c49007abf37cfd8a82b5e1c06214fc6fd" exitCode=0
Sep 17 16:21:49.903965 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.903774    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"
Sep 17 16:21:49.903965 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.903794    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" event={"ID":"edc71fd8-61bc-43ad-9e72-591c0959860b","Type":"ContainerDied","Data":"04a11eb05d1961dc895367323fc7245c49007abf37cfd8a82b5e1c06214fc6fd"}
Sep 17 16:21:49.903965 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.903846    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq" event={"ID":"edc71fd8-61bc-43ad-9e72-591c0959860b","Type":"ContainerDied","Data":"a653d1c9a1454f0c6e6fcdf7dd809c2fe741b105ac1c7a1d4ee776beef04c98d"}
Sep 17 16:21:49.903965 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.903869    2574 scope.go:117] "RemoveContainer" containerID="04a11eb05d1961dc895367323fc7245c49007abf37cfd8a82b5e1c06214fc6fd"
Sep 17 16:21:49.913188 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.913171    2574 scope.go:117] "RemoveContainer" containerID="91d14e35773b979462d5dacf7bb05aa932832b7e8d149501ecc8d62fd1428393"
Sep 17 16:21:49.921233 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.921216    2574 scope.go:117] "RemoveContainer" containerID="04a11eb05d1961dc895367323fc7245c49007abf37cfd8a82b5e1c06214fc6fd"
Sep 17 16:21:49.926802 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:21:49.921498    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a11eb05d1961dc895367323fc7245c49007abf37cfd8a82b5e1c06214fc6fd\": container with ID starting with 04a11eb05d1961dc895367323fc7245c49007abf37cfd8a82b5e1c06214fc6fd not found: ID does not exist" containerID="04a11eb05d1961dc895367323fc7245c49007abf37cfd8a82b5e1c06214fc6fd"
Sep 17 16:21:49.926802 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.921861    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a11eb05d1961dc895367323fc7245c49007abf37cfd8a82b5e1c06214fc6fd"} err="failed to get container status \"04a11eb05d1961dc895367323fc7245c49007abf37cfd8a82b5e1c06214fc6fd\": rpc error: code = NotFound desc = could not find container \"04a11eb05d1961dc895367323fc7245c49007abf37cfd8a82b5e1c06214fc6fd\": container with ID starting with 04a11eb05d1961dc895367323fc7245c49007abf37cfd8a82b5e1c06214fc6fd not found: ID does not exist"
Sep 17 16:21:49.926802 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.921888    2574 scope.go:117] "RemoveContainer" containerID="91d14e35773b979462d5dacf7bb05aa932832b7e8d149501ecc8d62fd1428393"
Sep 17 16:21:49.928862 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:21:49.927388    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d14e35773b979462d5dacf7bb05aa932832b7e8d149501ecc8d62fd1428393\": container with ID starting with 91d14e35773b979462d5dacf7bb05aa932832b7e8d149501ecc8d62fd1428393 not found: ID does not exist" containerID="91d14e35773b979462d5dacf7bb05aa932832b7e8d149501ecc8d62fd1428393"
Sep 17 16:21:49.928986 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.927431    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d14e35773b979462d5dacf7bb05aa932832b7e8d149501ecc8d62fd1428393"} err="failed to get container status \"91d14e35773b979462d5dacf7bb05aa932832b7e8d149501ecc8d62fd1428393\": rpc error: code = NotFound desc = could not find container \"91d14e35773b979462d5dacf7bb05aa932832b7e8d149501ecc8d62fd1428393\": container with ID starting with 91d14e35773b979462d5dacf7bb05aa932832b7e8d149501ecc8d62fd1428393 not found: ID does not exist"
Sep 17 16:21:49.930175 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.930147    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"]
Sep 17 16:21:49.931745 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:49.931726    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b5fb95f4d-krcfq"]
Sep 17 16:21:51.049576 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:51.049545    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" path="/var/lib/kubelet/pods/edc71fd8-61bc-43ad-9e72-591c0959860b/volumes"
Sep 17 16:21:52.768699 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.768669    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"]
Sep 17 16:21:52.769095 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.769049    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerName="storage-initializer"
Sep 17 16:21:52.769095 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.769061    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerName="storage-initializer"
Sep 17 16:21:52.769095 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.769069    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="storage-initializer"
Sep 17 16:21:52.769095 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.769074    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="storage-initializer"
Sep 17 16:21:52.769095 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.769082    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main"
Sep 17 16:21:52.769095 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.769088    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main"
Sep 17 16:21:52.769326 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.769102    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerName="main"
Sep 17 16:21:52.769326 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.769108    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerName="main"
Sep 17 16:21:52.769326 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.769163    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc71fd8-61bc-43ad-9e72-591c0959860b" containerName="main"
Sep 17 16:21:52.769326 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.769171    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c193b381-0c6e-4cfe-b82e-8c351b10e126" containerName="main"
Sep 17 16:21:52.774097 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.774077    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.776399 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.776371    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 16:21:52.776545 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.776458    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 17 16:21:52.776545 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.776490    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 16:21:52.776545 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.776465    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:21:52.776719 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.776641    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 16:21:52.776769 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.776731    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-472dh\""
Sep 17 16:21:52.787613 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.787591    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"]
Sep 17 16:21:52.789727 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.789700    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:52.822679 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.822652    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-model-cache\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.822817 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.822687    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-home\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.822817 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.822710    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mv6t\" (UniqueName: \"kubernetes.io/projected/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-kube-api-access-4mv6t\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.822817 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.822733    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-kserve-provision-location\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.822930 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.822810    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-tmp-dir\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.822930 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.822848    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-cabundle-cert\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.822930 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.822871    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-tls-certs\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.823028 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.822936    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-dshm\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.923673 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.923640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-model-cache\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.923874 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.923681    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-home\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.923874 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.923707    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mv6t\" (UniqueName: \"kubernetes.io/projected/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-kube-api-access-4mv6t\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.923874 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.923744    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-kserve-provision-location\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.923874 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.923789    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-tmp-dir\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.923874 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.923824    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-cabundle-cert\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.923874 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.923848    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-tls-certs\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.924206 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.923885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-dshm\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.924206 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.924116    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-model-cache\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.924206 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.924134    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-home\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.924571 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.924546    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-tmp-dir\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.924692 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.924611    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-kserve-provision-location\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.924692 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.924628    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-cabundle-cert\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.926382 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.926363    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-dshm\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.926677 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.926658    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-tls-certs\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:52.931698 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:52.931675    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mv6t\" (UniqueName: \"kubernetes.io/projected/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-kube-api-access-4mv6t\") pod \"router-with-refs-test-kserve-7cc98c4c48-rxw7w\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:53.027826 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.027750    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"]
Sep 17 16:21:53.032323 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.032296    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"
Sep 17 16:21:53.034632 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.034594    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-zcwdr\""
Sep 17 16:21:53.050870 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.050842    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"]
Sep 17 16:21:53.078742 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.078707    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:53.084559 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.084533    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:21:53.125796 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.125235    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqpsl\" (UniqueName: \"kubernetes.io/projected/45edd319-67b9-4447-bd5e-33d867662ce1-kube-api-access-lqpsl\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6\" (UID: \"45edd319-67b9-4447-bd5e-33d867662ce1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"
Sep 17 16:21:53.125796 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.125288    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45edd319-67b9-4447-bd5e-33d867662ce1-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6\" (UID: \"45edd319-67b9-4447-bd5e-33d867662ce1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"
Sep 17 16:21:53.182044 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.182002    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:53.225978 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.225944    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqpsl\" (UniqueName: \"kubernetes.io/projected/45edd319-67b9-4447-bd5e-33d867662ce1-kube-api-access-lqpsl\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6\" (UID: \"45edd319-67b9-4447-bd5e-33d867662ce1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"
Sep 17 16:21:53.225978 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.225981    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45edd319-67b9-4447-bd5e-33d867662ce1-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6\" (UID: \"45edd319-67b9-4447-bd5e-33d867662ce1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"
Sep 17 16:21:53.228662 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.228630    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45edd319-67b9-4447-bd5e-33d867662ce1-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6\" (UID: \"45edd319-67b9-4447-bd5e-33d867662ce1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"
Sep 17 16:21:53.234343 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.234318    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqpsl\" (UniqueName: \"kubernetes.io/projected/45edd319-67b9-4447-bd5e-33d867662ce1-kube-api-access-lqpsl\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6\" (UID: \"45edd319-67b9-4447-bd5e-33d867662ce1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"
Sep 17 16:21:53.347397 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.347310    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"
Sep 17 16:21:53.366797 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.366764    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"]
Sep 17 16:21:53.368704 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:21:53.368667    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4124cbff_1da7_4f59_9b22_d4dbbed3c8b2.slice/crio-e2bf9eb27d9d849c8c7c7ffc1d0666009d661451b057ef628f8fc3c7d10bd89c WatchSource:0}: Error finding container e2bf9eb27d9d849c8c7c7ffc1d0666009d661451b057ef628f8fc3c7d10bd89c: Status 404 returned error can't find the container with id e2bf9eb27d9d849c8c7c7ffc1d0666009d661451b057ef628f8fc3c7d10bd89c
Sep 17 16:21:53.626663 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.626636    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"]
Sep 17 16:21:53.628747 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:21:53.628712    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45edd319_67b9_4447_bd5e_33d867662ce1.slice/crio-b857c85329136b119ee64366a7e0f6b262dfdc1f43a7f5b835b0359eda37a631 WatchSource:0}: Error finding container b857c85329136b119ee64366a7e0f6b262dfdc1f43a7f5b835b0359eda37a631: Status 404 returned error can't find the container with id b857c85329136b119ee64366a7e0f6b262dfdc1f43a7f5b835b0359eda37a631
Sep 17 16:21:53.929581 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.929482    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6" event={"ID":"45edd319-67b9-4447-bd5e-33d867662ce1","Type":"ContainerStarted","Data":"99e645fe884e98cc4f39e7fccb68c7a448854b256ca44700507244a650c007b1"}
Sep 17 16:21:53.930038 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.929595    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6" event={"ID":"45edd319-67b9-4447-bd5e-33d867662ce1","Type":"ContainerStarted","Data":"b857c85329136b119ee64366a7e0f6b262dfdc1f43a7f5b835b0359eda37a631"}
Sep 17 16:21:53.930038 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.929629    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"
Sep 17 16:21:53.931521 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.931486    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" event={"ID":"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2","Type":"ContainerStarted","Data":"385b533f609abf1bdfc9354783ea30ae206836d077459181ce903c1a1e3271c6"}
Sep 17 16:21:53.931641 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.931544    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" event={"ID":"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2","Type":"ContainerStarted","Data":"e2bf9eb27d9d849c8c7c7ffc1d0666009d661451b057ef628f8fc3c7d10bd89c"}
Sep 17 16:21:53.945994 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:53.945945    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6" podStartSLOduration=0.945929025 podStartE2EDuration="945.929025ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:21:53.944838996 +0000 UTC m=+2579.416678841" watchObservedRunningTime="2026-09-17 16:21:53.945929025 +0000 UTC m=+2579.417768870"
Sep 17 16:21:55.104759 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:55.104722    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:55.166581 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:55.166541    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:55.944013 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:55.943977    2574 generic.go:356] "Generic (PLEG): container finished" podID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerID="385b533f609abf1bdfc9354783ea30ae206836d077459181ce903c1a1e3271c6" exitCode=0
Sep 17 16:21:55.944185 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:55.944052    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" event={"ID":"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2","Type":"ContainerDied","Data":"385b533f609abf1bdfc9354783ea30ae206836d077459181ce903c1a1e3271c6"}
Sep 17 16:21:56.951547 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:56.951513    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" event={"ID":"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2","Type":"ContainerStarted","Data":"4954acae73ccbf6b0e632fe261f1a90ffc83080bcef1cba21288f94df6485b1f"}
Sep 17 16:21:56.975199 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:56.975139    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" podStartSLOduration=2.681145729 podStartE2EDuration="4.975122414s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.293976685s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:21:56.974045498 +0000 UTC m=+2582.445885334" watchObservedRunningTime="2026-09-17 16:21:56.975122414 +0000 UTC m=+2582.446962260"
Sep 17 16:21:57.084974 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:21:57.084935    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:22:03.085293 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:03.085253    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:22:03.086702 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:03.086664    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 16:22:13.085434 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:13.085382    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 16:22:23.085397 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:23.085295    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 16:22:23.882818 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:23.882749    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:23.989991 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:23.989941    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:24.093650 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:24.093615    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:24.939798 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:24.939768    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"
Sep 17 16:22:30.879944 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:30.879903    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:32.808170 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.808131    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"]
Sep 17 16:22:32.813465 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.813428    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.815922 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.815890    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 16:22:32.816084 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.815945    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:22:32.816084 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.815897    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-ns7z5\""
Sep 17 16:22:32.816084 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.815900    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 16:22:32.816084 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.815900    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 16:22:32.816267 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.815896    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 16:22:32.828032 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.828011    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"]
Sep 17 16:22:32.876989 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.876952    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:32.895757 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.895728    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-dshm\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.895906 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.895764    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-tmp-dir\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.895906 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.895792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn55x\" (UniqueName: \"kubernetes.io/projected/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-kube-api-access-gn55x\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.895906 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.895836    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-kserve-provision-location\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.895906 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.895855    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-home\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.895906 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.895882    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-cabundle-cert\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.896100 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.895967    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-model-cache\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.896100 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.896004    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-tls-certs\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.997137 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.997101    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-tls-certs\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.997349 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.997157    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-dshm\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.997349 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.997198    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-tmp-dir\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.997349 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.997241    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn55x\" (UniqueName: \"kubernetes.io/projected/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-kube-api-access-gn55x\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.997349 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.997269    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-kserve-provision-location\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.997349 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.997298    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-home\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.997349 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.997341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-cabundle-cert\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.997698 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.997391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-model-cache\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.997781 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.997745    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-tmp-dir\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.997849 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.997806    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-model-cache\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.998155 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.998129    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-cabundle-cert\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.998400 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.998380    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-kserve-provision-location\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.998507 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.998485    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-home\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.999639 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.999617    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-dshm\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:32.999753 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:32.999735    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-tls-certs\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:33.007660 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:33.007636    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn55x\" (UniqueName: \"kubernetes.io/projected/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-kube-api-access-gn55x\") pod \"stop-feature-test-kserve-85fbf97c6b-kx6nt\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:33.085041 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:33.084946    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 16:22:33.092105 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:33.092074    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:33.127891 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:33.127850    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:33.412188 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:33.412146    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"]
Sep 17 16:22:33.431151 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:22:33.431121    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4c0dfd_4d1f_4bab_b19f_454c7042a2cf.slice/crio-10eb53f9f68fd2f0e2b23ff7d6a64913c1bea4a6db3dae7270ec79243bab5f54 WatchSource:0}: Error finding container 10eb53f9f68fd2f0e2b23ff7d6a64913c1bea4a6db3dae7270ec79243bab5f54: Status 404 returned error can't find the container with id 10eb53f9f68fd2f0e2b23ff7d6a64913c1bea4a6db3dae7270ec79243bab5f54
Sep 17 16:22:34.142921 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:34.142883    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" event={"ID":"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf","Type":"ContainerStarted","Data":"382f3b155328762a2988cbb8363a9bb549ecb97c96e246ee73ec28d06b1e7b3d"}
Sep 17 16:22:34.143422 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:34.142938    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" event={"ID":"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf","Type":"ContainerStarted","Data":"10eb53f9f68fd2f0e2b23ff7d6a64913c1bea4a6db3dae7270ec79243bab5f54"}
Sep 17 16:22:36.155870 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:36.155829    2574 generic.go:356] "Generic (PLEG): container finished" podID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerID="382f3b155328762a2988cbb8363a9bb549ecb97c96e246ee73ec28d06b1e7b3d" exitCode=0
Sep 17 16:22:36.156287 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:36.155900    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" event={"ID":"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf","Type":"ContainerDied","Data":"382f3b155328762a2988cbb8363a9bb549ecb97c96e246ee73ec28d06b1e7b3d"}
Sep 17 16:22:37.163590 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:37.163558    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" event={"ID":"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf","Type":"ContainerStarted","Data":"4d2d8103bf182df9fe1359776bafe2ca023225071ae0d50827e87a7a49e1f2c1"}
Sep 17 16:22:37.841540 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:37.841501    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:37.919712 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:37.919674    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:38.128457 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:38.128391    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:43.085738 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:43.085682    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 16:22:43.128870 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:43.128829    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:22:43.130720 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:43.130689    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 17 16:22:46.229823 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:46.229776    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:53.085913 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:53.085858    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 16:22:53.129150 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:22:53.129103    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 17 16:23:03.086015 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:23:03.085966    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 16:23:03.128518 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:23:03.128475    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 17 16:23:13.085358 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:23:13.085298    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 16:23:13.129134 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:23:13.129082    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 17 16:23:23.085239 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:23:23.085187    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 16:23:23.128493 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:23:23.128426    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 17 16:23:33.086103 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:23:33.086049    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 16:23:33.128911 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:23:33.128864    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 17 16:23:43.085798 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:23:43.085752    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 16:23:43.128690 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:23:43.128636    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 17 16:23:53.095046 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:23:53.094957    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:23:53.102998 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:23:53.102966    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:23:53.122023 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:23:53.121952    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" podStartSLOduration=78.658757599 podStartE2EDuration="1m21.121918645s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.463161046s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:22:37.217050952 +0000 UTC m=+2622.688890825" watchObservedRunningTime="2026-09-17 16:23:53.121918645 +0000 UTC m=+2698.593758503"
Sep 17 16:23:53.129167 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:23:53.129137    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 17 16:24:00.686256 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:00.686218    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"]
Sep 17 16:24:00.689372 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:00.689326    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"]
Sep 17 16:24:00.690175 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:00.690145    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:00.779936 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:00.779891    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:03.129793 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:03.129745    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 17 16:24:04.939568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:04.939527    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6" podUID="45edd319-67b9-4447-bd5e-33d867662ce1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:24:13.128312 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:13.128265    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 17 16:24:14.939313 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:14.939259    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6" podUID="45edd319-67b9-4447-bd5e-33d867662ce1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:24:15.708475 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:15.708412    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6" podUID="45edd319-67b9-4447-bd5e-33d867662ce1" containerName="main" containerID="cri-o://99e645fe884e98cc4f39e7fccb68c7a448854b256ca44700507244a650c007b1" gracePeriod=45
Sep 17 16:24:15.709486 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:15.709420    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main" containerID="cri-o://4954acae73ccbf6b0e632fe261f1a90ffc83080bcef1cba21288f94df6485b1f" gracePeriod=45
Sep 17 16:24:16.096182 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.096078    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 17 16:24:16.724932 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.724893    2574 generic.go:356] "Generic (PLEG): container finished" podID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerID="4954acae73ccbf6b0e632fe261f1a90ffc83080bcef1cba21288f94df6485b1f" exitCode=0
Sep 17 16:24:16.725108 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.724964    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" event={"ID":"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2","Type":"ContainerDied","Data":"4954acae73ccbf6b0e632fe261f1a90ffc83080bcef1cba21288f94df6485b1f"}
Sep 17 16:24:16.765773 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.765749    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:24:16.896595 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.896560    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-model-cache\") pod \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") "
Sep 17 16:24:16.896772 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.896641    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mv6t\" (UniqueName: \"kubernetes.io/projected/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-kube-api-access-4mv6t\") pod \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") "
Sep 17 16:24:16.896772 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.896713    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-tmp-dir\") pod \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") "
Sep 17 16:24:16.896772 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.896747    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-home\") pod \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") "
Sep 17 16:24:16.896922 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.896770    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-cabundle-cert\") pod \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") "
Sep 17 16:24:16.896922 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.896794    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-dshm\") pod \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") "
Sep 17 16:24:16.896922 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.896841    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-tls-certs\") pod \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") "
Sep 17 16:24:16.896922 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.896870    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-kserve-provision-location\") pod \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\" (UID: \"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2\") "
Sep 17 16:24:16.896922 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.896886    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" (UID: "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:24:16.897225 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.897204    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:16.897297 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.897193    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" (UID: "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:24:16.897621 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.897589    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-home" (OuterVolumeSpecName: "home") pod "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" (UID: "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:24:16.899262 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.899237    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" (UID: "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:24:16.899509 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.899479    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-kube-api-access-4mv6t" (OuterVolumeSpecName: "kube-api-access-4mv6t") pod "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" (UID: "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2"). InnerVolumeSpecName "kube-api-access-4mv6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:24:16.899629 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.899606    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-dshm" (OuterVolumeSpecName: "dshm") pod "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" (UID: "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:24:16.912877 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.912845    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" (UID: "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:24:16.922337 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.922306    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" (UID: "4124cbff-1da7-4f59-9b22-d4dbbed3c8b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:24:16.998396 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.998345    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:16.998396 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.998390    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:16.998396 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.998401    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mv6t\" (UniqueName: \"kubernetes.io/projected/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-kube-api-access-4mv6t\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:16.998628 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.998411    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:16.998628 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.998419    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:16.998628 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.998429    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:16.998628 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:16.998449    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:17.731256 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:17.731218    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"
Sep 17 16:24:17.731256 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:17.731238    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w" event={"ID":"4124cbff-1da7-4f59-9b22-d4dbbed3c8b2","Type":"ContainerDied","Data":"e2bf9eb27d9d849c8c7c7ffc1d0666009d661451b057ef628f8fc3c7d10bd89c"}
Sep 17 16:24:17.731798 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:17.731315    2574 scope.go:117] "RemoveContainer" containerID="4954acae73ccbf6b0e632fe261f1a90ffc83080bcef1cba21288f94df6485b1f"
Sep 17 16:24:17.740914 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:17.740899    2574 scope.go:117] "RemoveContainer" containerID="385b533f609abf1bdfc9354783ea30ae206836d077459181ce903c1a1e3271c6"
Sep 17 16:24:17.750236 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:17.750207    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"]
Sep 17 16:24:17.755567 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:17.755545    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cc98c4c48-rxw7w"]
Sep 17 16:24:19.048617 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:19.048573    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" path="/var/lib/kubelet/pods/4124cbff-1da7-4f59-9b22-d4dbbed3c8b2/volumes"
Sep 17 16:24:23.128503 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:23.128455    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 17 16:24:24.940152 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:24.940111    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6" podUID="45edd319-67b9-4447-bd5e-33d867662ce1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:24:24.940657 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:24.940220    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"
Sep 17 16:24:33.138435 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:33.138342    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:24:33.146502 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:33.146470    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:24:34.940039 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:34.939992    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6" podUID="45edd319-67b9-4447-bd5e-33d867662ce1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:24:35.116331 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:35.116295    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"]
Sep 17 16:24:35.149084 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:35.149048    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:35.247110 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:35.247008    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:35.337756 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:35.337713    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:44.939197 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:44.939158    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6" podUID="45edd319-67b9-4447-bd5e-33d867662ce1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:24:45.881225 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:45.881194    2574 generic.go:356] "Generic (PLEG): container finished" podID="45edd319-67b9-4447-bd5e-33d867662ce1" containerID="99e645fe884e98cc4f39e7fccb68c7a448854b256ca44700507244a650c007b1" exitCode=0
Sep 17 16:24:45.881393 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:45.881246    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6" event={"ID":"45edd319-67b9-4447-bd5e-33d867662ce1","Type":"ContainerDied","Data":"99e645fe884e98cc4f39e7fccb68c7a448854b256ca44700507244a650c007b1"}
Sep 17 16:24:45.965726 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:45.965698    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"
Sep 17 16:24:46.063720 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:46.063645    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45edd319-67b9-4447-bd5e-33d867662ce1-tls-certs\") pod \"45edd319-67b9-4447-bd5e-33d867662ce1\" (UID: \"45edd319-67b9-4447-bd5e-33d867662ce1\") "
Sep 17 16:24:46.063720 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:46.063681    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqpsl\" (UniqueName: \"kubernetes.io/projected/45edd319-67b9-4447-bd5e-33d867662ce1-kube-api-access-lqpsl\") pod \"45edd319-67b9-4447-bd5e-33d867662ce1\" (UID: \"45edd319-67b9-4447-bd5e-33d867662ce1\") "
Sep 17 16:24:46.065753 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:46.065725    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45edd319-67b9-4447-bd5e-33d867662ce1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45edd319-67b9-4447-bd5e-33d867662ce1" (UID: "45edd319-67b9-4447-bd5e-33d867662ce1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:24:46.065863 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:46.065763    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45edd319-67b9-4447-bd5e-33d867662ce1-kube-api-access-lqpsl" (OuterVolumeSpecName: "kube-api-access-lqpsl") pod "45edd319-67b9-4447-bd5e-33d867662ce1" (UID: "45edd319-67b9-4447-bd5e-33d867662ce1"). InnerVolumeSpecName "kube-api-access-lqpsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:24:46.164922 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:46.164892    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45edd319-67b9-4447-bd5e-33d867662ce1-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:46.164922 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:46.164922    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqpsl\" (UniqueName: \"kubernetes.io/projected/45edd319-67b9-4447-bd5e-33d867662ce1-kube-api-access-lqpsl\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:46.888119 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:46.888084    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"
Sep 17 16:24:46.888119 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:46.888104    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6" event={"ID":"45edd319-67b9-4447-bd5e-33d867662ce1","Type":"ContainerDied","Data":"b857c85329136b119ee64366a7e0f6b262dfdc1f43a7f5b835b0359eda37a631"}
Sep 17 16:24:46.888357 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:46.888193    2574 scope.go:117] "RemoveContainer" containerID="99e645fe884e98cc4f39e7fccb68c7a448854b256ca44700507244a650c007b1"
Sep 17 16:24:46.914374 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:46.914339    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"]
Sep 17 16:24:46.922894 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:46.922870    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rszs6"]
Sep 17 16:24:47.049059 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:47.049030    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45edd319-67b9-4447-bd5e-33d867662ce1" path="/var/lib/kubelet/pods/45edd319-67b9-4447-bd5e-33d867662ce1/volumes"
Sep 17 16:24:50.138561 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:50.138505    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main" containerID="cri-o://4d2d8103bf182df9fe1359776bafe2ca023225071ae0d50827e87a7a49e1f2c1" gracePeriod=45
Sep 17 16:24:50.865323 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:50.865296    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:50.899787 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:50.899749    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:24:50.910789 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:50.910759    2574 generic.go:356] "Generic (PLEG): container finished" podID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerID="4d2d8103bf182df9fe1359776bafe2ca023225071ae0d50827e87a7a49e1f2c1" exitCode=0
Sep 17 16:24:50.910899 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:50.910802    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"
Sep 17 16:24:50.910899 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:50.910833    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" event={"ID":"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf","Type":"ContainerDied","Data":"4d2d8103bf182df9fe1359776bafe2ca023225071ae0d50827e87a7a49e1f2c1"}
Sep 17 16:24:50.910899 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:50.910886    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt" event={"ID":"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf","Type":"ContainerDied","Data":"10eb53f9f68fd2f0e2b23ff7d6a64913c1bea4a6db3dae7270ec79243bab5f54"}
Sep 17 16:24:50.911030 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:50.910908    2574 scope.go:117] "RemoveContainer" containerID="4d2d8103bf182df9fe1359776bafe2ca023225071ae0d50827e87a7a49e1f2c1"
Sep 17 16:24:50.920799 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:50.920775    2574 scope.go:117] "RemoveContainer" containerID="382f3b155328762a2988cbb8363a9bb549ecb97c96e246ee73ec28d06b1e7b3d"
Sep 17 16:24:50.930110 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:50.930093    2574 scope.go:117] "RemoveContainer" containerID="4d2d8103bf182df9fe1359776bafe2ca023225071ae0d50827e87a7a49e1f2c1"
Sep 17 16:24:50.930380 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:24:50.930351    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d2d8103bf182df9fe1359776bafe2ca023225071ae0d50827e87a7a49e1f2c1\": container with ID starting with 4d2d8103bf182df9fe1359776bafe2ca023225071ae0d50827e87a7a49e1f2c1 not found: ID does not exist" containerID="4d2d8103bf182df9fe1359776bafe2ca023225071ae0d50827e87a7a49e1f2c1"
Sep 17 16:24:50.930427 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:50.930390    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d2d8103bf182df9fe1359776bafe2ca023225071ae0d50827e87a7a49e1f2c1"} err="failed to get container status \"4d2d8103bf182df9fe1359776bafe2ca023225071ae0d50827e87a7a49e1f2c1\": rpc error: code = NotFound desc = could not find container \"4d2d8103bf182df9fe1359776bafe2ca023225071ae0d50827e87a7a49e1f2c1\": container with ID starting with 4d2d8103bf182df9fe1359776bafe2ca023225071ae0d50827e87a7a49e1f2c1 not found: ID does not exist"
Sep 17 16:24:50.930427 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:50.930407    2574 scope.go:117] "RemoveContainer" containerID="382f3b155328762a2988cbb8363a9bb549ecb97c96e246ee73ec28d06b1e7b3d"
Sep 17 16:24:50.930662 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:24:50.930647    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382f3b155328762a2988cbb8363a9bb549ecb97c96e246ee73ec28d06b1e7b3d\": container with ID starting with 382f3b155328762a2988cbb8363a9bb549ecb97c96e246ee73ec28d06b1e7b3d not found: ID does not exist" containerID="382f3b155328762a2988cbb8363a9bb549ecb97c96e246ee73ec28d06b1e7b3d"
Sep 17 16:24:50.930701 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:50.930664    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382f3b155328762a2988cbb8363a9bb549ecb97c96e246ee73ec28d06b1e7b3d"} err="failed to get container status \"382f3b155328762a2988cbb8363a9bb549ecb97c96e246ee73ec28d06b1e7b3d\": rpc error: code = NotFound desc = could not find container \"382f3b155328762a2988cbb8363a9bb549ecb97c96e246ee73ec28d06b1e7b3d\": container with ID starting with 382f3b155328762a2988cbb8363a9bb549ecb97c96e246ee73ec28d06b1e7b3d not found: ID does not exist"
Sep 17 16:24:51.008414 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.008382    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-home\") pod \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") "
Sep 17 16:24:51.008602 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.008426    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-dshm\") pod \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") "
Sep 17 16:24:51.008602 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.008487    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-model-cache\") pod \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") "
Sep 17 16:24:51.008602 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.008517    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-tls-certs\") pod \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") "
Sep 17 16:24:51.008602 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.008544    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-kserve-provision-location\") pod \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") "
Sep 17 16:24:51.008602 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.008602    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn55x\" (UniqueName: \"kubernetes.io/projected/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-kube-api-access-gn55x\") pod \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") "
Sep 17 16:24:51.008883 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.008628    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-tmp-dir\") pod \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") "
Sep 17 16:24:51.008883 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.008667    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-cabundle-cert\") pod \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\" (UID: \"7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf\") "
Sep 17 16:24:51.008883 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.008724    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" (UID: "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:24:51.009057 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.008977    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:51.009370 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.009342    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" (UID: "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:24:51.009539 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.009515    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-home" (OuterVolumeSpecName: "home") pod "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" (UID: "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:24:51.011194 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.011135    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" (UID: "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:24:51.011316 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.011196    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-dshm" (OuterVolumeSpecName: "dshm") pod "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" (UID: "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:24:51.011546 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.011518    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-kube-api-access-gn55x" (OuterVolumeSpecName: "kube-api-access-gn55x") pod "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" (UID: "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf"). InnerVolumeSpecName "kube-api-access-gn55x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:24:51.028327 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.028295    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" (UID: "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:24:51.036554 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.036527    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" (UID: "7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:24:51.054843 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.054803    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:51.110298 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.110265    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:51.110298 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.110298    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:51.110477 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.110339    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:51.110477 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.110358    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:51.110477 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.110373    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn55x\" (UniqueName: \"kubernetes.io/projected/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-kube-api-access-gn55x\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:51.110477 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.110386    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:51.110477 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.110400    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:24:51.194964 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.194891    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:51.236757 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.236726    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"]
Sep 17 16:24:51.260720 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:51.260687    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-kx6nt"]
Sep 17 16:24:53.048213 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:53.048182    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" path="/var/lib/kubelet/pods/7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf/volumes"
Sep 17 16:24:55.909144 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:24:55.909103    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:25:04.268883 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.268851    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"]
Sep 17 16:25:04.269333 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.269254    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="storage-initializer"
Sep 17 16:25:04.269333 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.269268    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="storage-initializer"
Sep 17 16:25:04.269333 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.269281    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="storage-initializer"
Sep 17 16:25:04.269333 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.269287    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="storage-initializer"
Sep 17 16:25:04.269333 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.269295    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main"
Sep 17 16:25:04.269333 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.269301    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main"
Sep 17 16:25:04.269333 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.269312    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main"
Sep 17 16:25:04.269333 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.269317    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main"
Sep 17 16:25:04.269333 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.269325    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45edd319-67b9-4447-bd5e-33d867662ce1" containerName="main"
Sep 17 16:25:04.269333 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.269330    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="45edd319-67b9-4447-bd5e-33d867662ce1" containerName="main"
Sep 17 16:25:04.269679 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.269392    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="45edd319-67b9-4447-bd5e-33d867662ce1" containerName="main"
Sep 17 16:25:04.269679 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.269401    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e4c0dfd-4d1f-4bab-b19f-454c7042a2cf" containerName="main"
Sep 17 16:25:04.269679 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.269410    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4124cbff-1da7-4f59-9b22-d4dbbed3c8b2" containerName="main"
Sep 17 16:25:04.274154 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.274133    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.280261 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.280235    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 16:25:04.280479 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.280260    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-ns7z5\""
Sep 17 16:25:04.280479 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.280314    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:25:04.280479 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.280369    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 16:25:04.280479 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.280432    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 16:25:04.280745 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.280729    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 16:25:04.311687 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.311647    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"]
Sep 17 16:25:04.429846 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.429808    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-tmp-dir\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.429846 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.429851    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-dshm\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.430067 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.429867    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68434d9-c908-45f9-8683-9de8ec31ee81-tls-certs\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.430067 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.429939    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9rd8\" (UniqueName: \"kubernetes.io/projected/a68434d9-c908-45f9-8683-9de8ec31ee81-kube-api-access-s9rd8\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.430067 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.430039    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68434d9-c908-45f9-8683-9de8ec31ee81-cabundle-cert\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.430189 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.430071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-model-cache\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.430189 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.430091    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-home\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.430189 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.430151    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-kserve-provision-location\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.465057 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.465020    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:25:04.531336 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.531246    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-kserve-provision-location\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.531336 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.531290    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-tmp-dir\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.531336 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.531320    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-dshm\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.531672 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.531342    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68434d9-c908-45f9-8683-9de8ec31ee81-tls-certs\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.531672 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.531380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9rd8\" (UniqueName: \"kubernetes.io/projected/a68434d9-c908-45f9-8683-9de8ec31ee81-kube-api-access-s9rd8\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.531672 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.531481    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68434d9-c908-45f9-8683-9de8ec31ee81-cabundle-cert\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.531672 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.531514    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-model-cache\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.531672 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.531537    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-home\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.531942 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.531692    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-kserve-provision-location\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.531942 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.531789    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-tmp-dir\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.531942 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.531889    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-home\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.531942 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.531938    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-model-cache\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.532287 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.532261    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68434d9-c908-45f9-8683-9de8ec31ee81-cabundle-cert\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.533851 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.533829    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-dshm\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.534073 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.534056    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68434d9-c908-45f9-8683-9de8ec31ee81-tls-certs\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.545511 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.545487    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9rd8\" (UniqueName: \"kubernetes.io/projected/a68434d9-c908-45f9-8683-9de8ec31ee81-kube-api-access-s9rd8\") pod \"stop-feature-test-kserve-85fbf97c6b-xpqn7\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.584588 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.584559    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:04.805981 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.805894    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"]
Sep 17 16:25:04.809001 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.808978    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"
Sep 17 16:25:04.814560 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.814537    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-sggtj\""
Sep 17 16:25:04.832942 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.832914    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"]
Sep 17 16:25:04.898071 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.897769    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"]
Sep 17 16:25:04.902089 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:25:04.902062    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68434d9_c908_45f9_8683_9de8ec31ee81.slice/crio-79b17b311cb6adb5231fa6eaeaf38376ba2dc4b37e8e589f4ef901ec86183905 WatchSource:0}: Error finding container 79b17b311cb6adb5231fa6eaeaf38376ba2dc4b37e8e589f4ef901ec86183905: Status 404 returned error can't find the container with id 79b17b311cb6adb5231fa6eaeaf38376ba2dc4b37e8e589f4ef901ec86183905
Sep 17 16:25:04.928097 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.928071    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:25:04.936186 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.936156    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b116df-f80b-4753-91db-ac4439925715-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz\" (UID: \"85b116df-f80b-4753-91db-ac4439925715\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"
Sep 17 16:25:04.936290 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.936198    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8qj6\" (UniqueName: \"kubernetes.io/projected/85b116df-f80b-4753-91db-ac4439925715-kube-api-access-z8qj6\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz\" (UID: \"85b116df-f80b-4753-91db-ac4439925715\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"
Sep 17 16:25:04.984388 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.984353    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" event={"ID":"a68434d9-c908-45f9-8683-9de8ec31ee81","Type":"ContainerStarted","Data":"594bd7def420508cbaf9d248e8641e3a700ba753e4ea2904ad213fb7d8cb14df"}
Sep 17 16:25:04.984558 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:04.984414    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" event={"ID":"a68434d9-c908-45f9-8683-9de8ec31ee81","Type":"ContainerStarted","Data":"79b17b311cb6adb5231fa6eaeaf38376ba2dc4b37e8e589f4ef901ec86183905"}
Sep 17 16:25:05.037029 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:05.036995    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b116df-f80b-4753-91db-ac4439925715-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz\" (UID: \"85b116df-f80b-4753-91db-ac4439925715\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"
Sep 17 16:25:05.037224 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:05.037037    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8qj6\" (UniqueName: \"kubernetes.io/projected/85b116df-f80b-4753-91db-ac4439925715-kube-api-access-z8qj6\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz\" (UID: \"85b116df-f80b-4753-91db-ac4439925715\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"
Sep 17 16:25:05.039549 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:05.039525    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b116df-f80b-4753-91db-ac4439925715-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz\" (UID: \"85b116df-f80b-4753-91db-ac4439925715\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"
Sep 17 16:25:05.051150 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:05.051119    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8qj6\" (UniqueName: \"kubernetes.io/projected/85b116df-f80b-4753-91db-ac4439925715-kube-api-access-z8qj6\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz\" (UID: \"85b116df-f80b-4753-91db-ac4439925715\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"
Sep 17 16:25:05.120237 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:05.120199    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"
Sep 17 16:25:05.410280 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:05.410019    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"]
Sep 17 16:25:05.412417 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:25:05.412383    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b116df_f80b_4753_91db_ac4439925715.slice/crio-cc363a678c4afcb55acba7e844cb9d89a11a5067b4d1542c95338dc5bf76fd4a WatchSource:0}: Error finding container cc363a678c4afcb55acba7e844cb9d89a11a5067b4d1542c95338dc5bf76fd4a: Status 404 returned error can't find the container with id cc363a678c4afcb55acba7e844cb9d89a11a5067b4d1542c95338dc5bf76fd4a
Sep 17 16:25:05.990596 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:05.990562    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz" event={"ID":"85b116df-f80b-4753-91db-ac4439925715","Type":"ContainerStarted","Data":"6abd63b5dd7061b9e98d5c0099e076e81df030412e676b63514d41eb102d4bdf"}
Sep 17 16:25:05.990787 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:05.990617    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz" event={"ID":"85b116df-f80b-4753-91db-ac4439925715","Type":"ContainerStarted","Data":"cc363a678c4afcb55acba7e844cb9d89a11a5067b4d1542c95338dc5bf76fd4a"}
Sep 17 16:25:06.018994 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:06.018927    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz" podStartSLOduration=2.018903253 podStartE2EDuration="2.018903253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:25:06.014571925 +0000 UTC m=+2771.486411772" watchObservedRunningTime="2026-09-17 16:25:06.018903253 +0000 UTC m=+2771.490743099"
Sep 17 16:25:06.994732 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:06.994700    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"
Sep 17 16:25:08.001275 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:08.001236    2574 generic.go:356] "Generic (PLEG): container finished" podID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerID="594bd7def420508cbaf9d248e8641e3a700ba753e4ea2904ad213fb7d8cb14df" exitCode=0
Sep 17 16:25:08.001799 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:08.001306    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" event={"ID":"a68434d9-c908-45f9-8683-9de8ec31ee81","Type":"ContainerDied","Data":"594bd7def420508cbaf9d248e8641e3a700ba753e4ea2904ad213fb7d8cb14df"}
Sep 17 16:25:09.009348 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:09.009305    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" event={"ID":"a68434d9-c908-45f9-8683-9de8ec31ee81","Type":"ContainerStarted","Data":"afd49c788c354023a6b120aac7025fd0ac3b8786174d2cab6bc9c619cef99726"}
Sep 17 16:25:09.047427 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:09.046553    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podStartSLOduration=2.703084033 podStartE2EDuration="5.046530453s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.34344642s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:25:09.044922653 +0000 UTC m=+2774.516762499" watchObservedRunningTime="2026-09-17 16:25:09.046530453 +0000 UTC m=+2774.518370299"
Sep 17 16:25:09.584764 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:09.584723    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:14.585722 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:14.585686    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:25:14.587397 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:14.587365    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 16:25:17.769163 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:17.769076    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:25:17.998827 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:17.998796    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:25:20.330945 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:20.330911    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:25:24.581415 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.581384    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"]
Sep 17 16:25:24.585148 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.585124    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.585426 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.585388    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 16:25:24.589812 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.589409    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 16:25:24.589812 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.589484    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 16:25:24.589812 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.589613    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:25:24.590073 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.589910    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-gf264\""
Sep 17 16:25:24.590135 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.590101    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-2sg2q\""
Sep 17 16:25:24.590373 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.590353    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 16:25:24.590510 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.590363    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 16:25:24.606742 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.606716    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"]
Sep 17 16:25:24.616284 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.616258    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.616423 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.616306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.616423 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.616382    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjb69\" (UniqueName: \"kubernetes.io/projected/6a404d1a-5e27-40b6-b2ef-86a830393919-kube-api-access-kjb69\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.616555 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.616424    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.616555 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.616528    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.616657 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.616554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a404d1a-5e27-40b6-b2ef-86a830393919-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.616657 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.616585    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a404d1a-5e27-40b6-b2ef-86a830393919-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.616757 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.616654    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.645638 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.645603    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:25:24.717239 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.717208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.717239 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.717245    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a404d1a-5e27-40b6-b2ef-86a830393919-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.717551 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.717288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a404d1a-5e27-40b6-b2ef-86a830393919-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.717551 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.717323    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.717551 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.717392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.717551 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.717436    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.717551 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.717503    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjb69\" (UniqueName: \"kubernetes.io/projected/6a404d1a-5e27-40b6-b2ef-86a830393919-kube-api-access-kjb69\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.717551 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.717536    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.717869 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.717793    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.717928 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.717868    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.718095 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.718062    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.718185 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.718105    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a404d1a-5e27-40b6-b2ef-86a830393919-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.718247 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.718200    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.719763 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.719739    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.720033 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.720014    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a404d1a-5e27-40b6-b2ef-86a830393919-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.729723 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.729693    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjb69\" (UniqueName: \"kubernetes.io/projected/6a404d1a-5e27-40b6-b2ef-86a830393919-kube-api-access-kjb69\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.896828 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.896791    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:24.972567 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:24.972521    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:25:25.183944 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:25.183915    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"]
Sep 17 16:25:25.185461 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:25:25.185408    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a404d1a_5e27_40b6_b2ef_86a830393919.slice/crio-a32440549e75d3b938d7c0122e42474320358e9124b325a12866ab77db2e1f0c WatchSource:0}: Error finding container a32440549e75d3b938d7c0122e42474320358e9124b325a12866ab77db2e1f0c: Status 404 returned error can't find the container with id a32440549e75d3b938d7c0122e42474320358e9124b325a12866ab77db2e1f0c
Sep 17 16:25:26.104537 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:26.104500    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" event={"ID":"6a404d1a-5e27-40b6-b2ef-86a830393919","Type":"ContainerStarted","Data":"051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f"}
Sep 17 16:25:26.104962 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:26.104566    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" event={"ID":"6a404d1a-5e27-40b6-b2ef-86a830393919","Type":"ContainerStarted","Data":"a32440549e75d3b938d7c0122e42474320358e9124b325a12866ab77db2e1f0c"}
Sep 17 16:25:26.104962 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:26.104594    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:27.112557 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:27.112516    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" event={"ID":"6a404d1a-5e27-40b6-b2ef-86a830393919","Type":"ContainerStarted","Data":"338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48"}
Sep 17 16:25:27.707064 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:27.707018    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:25:29.127985 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:29.127950    2574 generic.go:356] "Generic (PLEG): container finished" podID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerID="338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48" exitCode=0
Sep 17 16:25:29.128426 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:29.128021    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" event={"ID":"6a404d1a-5e27-40b6-b2ef-86a830393919","Type":"ContainerDied","Data":"338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48"}
Sep 17 16:25:30.135888 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:30.135851    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" event={"ID":"6a404d1a-5e27-40b6-b2ef-86a830393919","Type":"ContainerStarted","Data":"240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3"}
Sep 17 16:25:34.586391 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:34.586336    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 16:25:34.897400 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:34.897367    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:34.897630 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:34.897478    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:34.898814 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:34.898778    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 16:25:38.003506 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:38.003471    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"
Sep 17 16:25:38.028515 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:38.028462    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podStartSLOduration=11.506775682 podStartE2EDuration="14.028428196s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.521652514s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:25:30.204414819 +0000 UTC m=+2795.676254666" watchObservedRunningTime="2026-09-17 16:25:38.028428196 +0000 UTC m=+2803.500268045"
Sep 17 16:25:39.267095 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:39.267060    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:25:44.585672 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:44.585616    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 16:25:44.898064 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:44.898014    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 16:25:44.916983 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:44.916947    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:25:44.955476 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:44.955417    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:25:45.035267 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:45.035228    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:25:51.664461 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:51.664398    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:25:54.585794 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:54.585753    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 16:25:54.897429 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:25:54.897371    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 16:26:04.493646 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:26:04.493598    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:26:04.585401 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:26:04.585343    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 16:26:04.593806 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:26:04.593770    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:26:04.898167 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:26:04.898119    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 16:26:14.585344 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:26:14.585293    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 16:26:14.897909 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:26:14.897860    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 16:26:24.585550 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:26:24.585500    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 16:26:24.897898 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:26:24.897849    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 16:26:34.585362 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:26:34.585300    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 16:26:34.897936 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:26:34.897882    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 16:26:44.585883 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:26:44.585833    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 16:26:44.897321 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:26:44.897262    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 16:26:54.585745 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:26:54.585641    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 16:26:54.897199 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:26:54.897144    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 16:27:04.585453 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:04.585393    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 16:27:04.897157 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:04.897102    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 16:27:14.585953 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:14.585904    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 16:27:14.897161 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:14.897101    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 16:27:24.596304 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:24.596269    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:27:24.605044 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:24.605017    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:27:24.897129 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:24.897089    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 16:27:26.687924 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:26.687886    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:27:26.716560 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:26.716527    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"]
Sep 17 16:27:26.728406 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:26.728375    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"]
Sep 17 16:27:27.879093 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:27.879032    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:27:28.003753 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:28.003717    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz" podUID="85b116df-f80b-4753-91db-ac4439925715" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:27:34.897858 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:34.897809    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 17 16:27:38.003272 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:38.003229    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz" podUID="85b116df-f80b-4753-91db-ac4439925715" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:27:41.737821 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:41.737776    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz" podUID="85b116df-f80b-4753-91db-ac4439925715" containerName="main" containerID="cri-o://6abd63b5dd7061b9e98d5c0099e076e81df030412e676b63514d41eb102d4bdf" gracePeriod=45
Sep 17 16:27:41.751452 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:41.751388    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main" containerID="cri-o://afd49c788c354023a6b120aac7025fd0ac3b8786174d2cab6bc9c619cef99726" gracePeriod=45
Sep 17 16:27:42.513133 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.513105    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:27:42.613774 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.613744    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68434d9-c908-45f9-8683-9de8ec31ee81-cabundle-cert\") pod \"a68434d9-c908-45f9-8683-9de8ec31ee81\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") "
Sep 17 16:27:42.613955 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.613812    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-model-cache\") pod \"a68434d9-c908-45f9-8683-9de8ec31ee81\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") "
Sep 17 16:27:42.613955 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.613882    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9rd8\" (UniqueName: \"kubernetes.io/projected/a68434d9-c908-45f9-8683-9de8ec31ee81-kube-api-access-s9rd8\") pod \"a68434d9-c908-45f9-8683-9de8ec31ee81\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") "
Sep 17 16:27:42.613955 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.613919    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-tmp-dir\") pod \"a68434d9-c908-45f9-8683-9de8ec31ee81\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") "
Sep 17 16:27:42.613955 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.613945    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-kserve-provision-location\") pod \"a68434d9-c908-45f9-8683-9de8ec31ee81\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") "
Sep 17 16:27:42.614181 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.613989    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-dshm\") pod \"a68434d9-c908-45f9-8683-9de8ec31ee81\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") "
Sep 17 16:27:42.614181 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.614030    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68434d9-c908-45f9-8683-9de8ec31ee81-tls-certs\") pod \"a68434d9-c908-45f9-8683-9de8ec31ee81\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") "
Sep 17 16:27:42.614181 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.614051    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-home\") pod \"a68434d9-c908-45f9-8683-9de8ec31ee81\" (UID: \"a68434d9-c908-45f9-8683-9de8ec31ee81\") "
Sep 17 16:27:42.614181 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.614120    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-model-cache" (OuterVolumeSpecName: "model-cache") pod "a68434d9-c908-45f9-8683-9de8ec31ee81" (UID: "a68434d9-c908-45f9-8683-9de8ec31ee81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:27:42.614394 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.614197    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68434d9-c908-45f9-8683-9de8ec31ee81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a68434d9-c908-45f9-8683-9de8ec31ee81" (UID: "a68434d9-c908-45f9-8683-9de8ec31ee81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:27:42.614481 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.614426    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:27:42.614481 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.614465    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68434d9-c908-45f9-8683-9de8ec31ee81-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:27:42.614862 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.614837    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-home" (OuterVolumeSpecName: "home") pod "a68434d9-c908-45f9-8683-9de8ec31ee81" (UID: "a68434d9-c908-45f9-8683-9de8ec31ee81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:27:42.616431 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.616385    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68434d9-c908-45f9-8683-9de8ec31ee81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a68434d9-c908-45f9-8683-9de8ec31ee81" (UID: "a68434d9-c908-45f9-8683-9de8ec31ee81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:27:42.616661 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.616585    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68434d9-c908-45f9-8683-9de8ec31ee81-kube-api-access-s9rd8" (OuterVolumeSpecName: "kube-api-access-s9rd8") pod "a68434d9-c908-45f9-8683-9de8ec31ee81" (UID: "a68434d9-c908-45f9-8683-9de8ec31ee81"). InnerVolumeSpecName "kube-api-access-s9rd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:27:42.617032 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.617011    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-dshm" (OuterVolumeSpecName: "dshm") pod "a68434d9-c908-45f9-8683-9de8ec31ee81" (UID: "a68434d9-c908-45f9-8683-9de8ec31ee81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:27:42.625851 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.625821    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a68434d9-c908-45f9-8683-9de8ec31ee81" (UID: "a68434d9-c908-45f9-8683-9de8ec31ee81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:27:42.640187 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.640163    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a68434d9-c908-45f9-8683-9de8ec31ee81" (UID: "a68434d9-c908-45f9-8683-9de8ec31ee81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:27:42.714862 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.714828    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:27:42.714862 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.714858    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:27:42.714862 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.714869    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:27:42.715095 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.714878    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68434d9-c908-45f9-8683-9de8ec31ee81-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:27:42.715095 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.714886    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68434d9-c908-45f9-8683-9de8ec31ee81-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:27:42.715095 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.714894    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9rd8\" (UniqueName: \"kubernetes.io/projected/a68434d9-c908-45f9-8683-9de8ec31ee81-kube-api-access-s9rd8\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:27:42.902482 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.902367    2574 generic.go:356] "Generic (PLEG): container finished" podID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerID="afd49c788c354023a6b120aac7025fd0ac3b8786174d2cab6bc9c619cef99726" exitCode=0
Sep 17 16:27:42.902482 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.902464    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"
Sep 17 16:27:42.902936 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.902482    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" event={"ID":"a68434d9-c908-45f9-8683-9de8ec31ee81","Type":"ContainerDied","Data":"afd49c788c354023a6b120aac7025fd0ac3b8786174d2cab6bc9c619cef99726"}
Sep 17 16:27:42.902936 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.902574    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7" event={"ID":"a68434d9-c908-45f9-8683-9de8ec31ee81","Type":"ContainerDied","Data":"79b17b311cb6adb5231fa6eaeaf38376ba2dc4b37e8e589f4ef901ec86183905"}
Sep 17 16:27:42.902936 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.902599    2574 scope.go:117] "RemoveContainer" containerID="afd49c788c354023a6b120aac7025fd0ac3b8786174d2cab6bc9c619cef99726"
Sep 17 16:27:42.912029 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.912011    2574 scope.go:117] "RemoveContainer" containerID="594bd7def420508cbaf9d248e8641e3a700ba753e4ea2904ad213fb7d8cb14df"
Sep 17 16:27:42.920284 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.920263    2574 scope.go:117] "RemoveContainer" containerID="afd49c788c354023a6b120aac7025fd0ac3b8786174d2cab6bc9c619cef99726"
Sep 17 16:27:42.920595 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:27:42.920577    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd49c788c354023a6b120aac7025fd0ac3b8786174d2cab6bc9c619cef99726\": container with ID starting with afd49c788c354023a6b120aac7025fd0ac3b8786174d2cab6bc9c619cef99726 not found: ID does not exist" containerID="afd49c788c354023a6b120aac7025fd0ac3b8786174d2cab6bc9c619cef99726"
Sep 17 16:27:42.920646 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.920604    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd49c788c354023a6b120aac7025fd0ac3b8786174d2cab6bc9c619cef99726"} err="failed to get container status \"afd49c788c354023a6b120aac7025fd0ac3b8786174d2cab6bc9c619cef99726\": rpc error: code = NotFound desc = could not find container \"afd49c788c354023a6b120aac7025fd0ac3b8786174d2cab6bc9c619cef99726\": container with ID starting with afd49c788c354023a6b120aac7025fd0ac3b8786174d2cab6bc9c619cef99726 not found: ID does not exist"
Sep 17 16:27:42.920646 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.920621    2574 scope.go:117] "RemoveContainer" containerID="594bd7def420508cbaf9d248e8641e3a700ba753e4ea2904ad213fb7d8cb14df"
Sep 17 16:27:42.920843 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:27:42.920829    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594bd7def420508cbaf9d248e8641e3a700ba753e4ea2904ad213fb7d8cb14df\": container with ID starting with 594bd7def420508cbaf9d248e8641e3a700ba753e4ea2904ad213fb7d8cb14df not found: ID does not exist" containerID="594bd7def420508cbaf9d248e8641e3a700ba753e4ea2904ad213fb7d8cb14df"
Sep 17 16:27:42.920887 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.920845    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594bd7def420508cbaf9d248e8641e3a700ba753e4ea2904ad213fb7d8cb14df"} err="failed to get container status \"594bd7def420508cbaf9d248e8641e3a700ba753e4ea2904ad213fb7d8cb14df\": rpc error: code = NotFound desc = could not find container \"594bd7def420508cbaf9d248e8641e3a700ba753e4ea2904ad213fb7d8cb14df\": container with ID starting with 594bd7def420508cbaf9d248e8641e3a700ba753e4ea2904ad213fb7d8cb14df not found: ID does not exist"
Sep 17 16:27:42.949976 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.949945    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"]
Sep 17 16:27:42.954111 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:42.954035    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85fbf97c6b-xpqn7"]
Sep 17 16:27:43.049005 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:43.048976    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" path="/var/lib/kubelet/pods/a68434d9-c908-45f9-8683-9de8ec31ee81/volumes"
Sep 17 16:27:44.907227 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:44.907192    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:27:44.920095 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:44.920068    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:27:48.003559 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:48.003517    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz" podUID="85b116df-f80b-4753-91db-ac4439925715" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:27:48.003948 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:48.003645    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"
Sep 17 16:27:54.172992 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:54.172945    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:27:54.213771 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:54.213731    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"]
Sep 17 16:27:54.297628 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:54.297584    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:27:58.003130 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:27:58.003093    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz" podUID="85b116df-f80b-4753-91db-ac4439925715" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:28:08.003236 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:08.003192    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz" podUID="85b116df-f80b-4753-91db-ac4439925715" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:28:09.237484 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:09.237400    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main" containerID="cri-o://240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3" gracePeriod=45
Sep 17 16:28:09.773771 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:09.773739    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="llm-d-routing-sidecar" containerID="cri-o://051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f" gracePeriod=45
Sep 17 16:28:10.017590 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.017563    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:28:10.040879 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.040842    2574 generic.go:356] "Generic (PLEG): container finished" podID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerID="240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3" exitCode=0
Sep 17 16:28:10.040879 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.040877    2574 generic.go:356] "Generic (PLEG): container finished" podID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerID="051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f" exitCode=0
Sep 17 16:28:10.041057 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.040877    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" event={"ID":"6a404d1a-5e27-40b6-b2ef-86a830393919","Type":"ContainerDied","Data":"240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3"}
Sep 17 16:28:10.041057 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.040902    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"
Sep 17 16:28:10.041057 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.040931    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" event={"ID":"6a404d1a-5e27-40b6-b2ef-86a830393919","Type":"ContainerDied","Data":"051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f"}
Sep 17 16:28:10.041057 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.040950    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns" event={"ID":"6a404d1a-5e27-40b6-b2ef-86a830393919","Type":"ContainerDied","Data":"a32440549e75d3b938d7c0122e42474320358e9124b325a12866ab77db2e1f0c"}
Sep 17 16:28:10.041057 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.040970    2574 scope.go:117] "RemoveContainer" containerID="240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3"
Sep 17 16:28:10.050145 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.050118    2574 scope.go:117] "RemoveContainer" containerID="338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48"
Sep 17 16:28:10.058741 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.058722    2574 scope.go:117] "RemoveContainer" containerID="051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f"
Sep 17 16:28:10.059172 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.059155    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-dshm\") pod \"6a404d1a-5e27-40b6-b2ef-86a830393919\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") "
Sep 17 16:28:10.059259 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.059216    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-tmp-dir\") pod \"6a404d1a-5e27-40b6-b2ef-86a830393919\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") "
Sep 17 16:28:10.059259 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.059239    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-model-cache\") pod \"6a404d1a-5e27-40b6-b2ef-86a830393919\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") "
Sep 17 16:28:10.059378 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.059289    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-home\") pod \"6a404d1a-5e27-40b6-b2ef-86a830393919\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") "
Sep 17 16:28:10.059378 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.059311    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a404d1a-5e27-40b6-b2ef-86a830393919-cabundle-cert\") pod \"6a404d1a-5e27-40b6-b2ef-86a830393919\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") "
Sep 17 16:28:10.059378 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.059337    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-kserve-provision-location\") pod \"6a404d1a-5e27-40b6-b2ef-86a830393919\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") "
Sep 17 16:28:10.059378 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.059373    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjb69\" (UniqueName: \"kubernetes.io/projected/6a404d1a-5e27-40b6-b2ef-86a830393919-kube-api-access-kjb69\") pod \"6a404d1a-5e27-40b6-b2ef-86a830393919\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") "
Sep 17 16:28:10.059758 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.059404    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a404d1a-5e27-40b6-b2ef-86a830393919-tls-certs\") pod \"6a404d1a-5e27-40b6-b2ef-86a830393919\" (UID: \"6a404d1a-5e27-40b6-b2ef-86a830393919\") "
Sep 17 16:28:10.060064 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.060000    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a404d1a-5e27-40b6-b2ef-86a830393919-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a404d1a-5e27-40b6-b2ef-86a830393919" (UID: "6a404d1a-5e27-40b6-b2ef-86a830393919"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:28:10.060366 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.060335    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-home" (OuterVolumeSpecName: "home") pod "6a404d1a-5e27-40b6-b2ef-86a830393919" (UID: "6a404d1a-5e27-40b6-b2ef-86a830393919"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:28:10.060692 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.060653    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a404d1a-5e27-40b6-b2ef-86a830393919" (UID: "6a404d1a-5e27-40b6-b2ef-86a830393919"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:28:10.061911 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.061868    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-dshm" (OuterVolumeSpecName: "dshm") pod "6a404d1a-5e27-40b6-b2ef-86a830393919" (UID: "6a404d1a-5e27-40b6-b2ef-86a830393919"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:28:10.062163 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.062106    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a404d1a-5e27-40b6-b2ef-86a830393919-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a404d1a-5e27-40b6-b2ef-86a830393919" (UID: "6a404d1a-5e27-40b6-b2ef-86a830393919"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:28:10.062561 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.062526    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a404d1a-5e27-40b6-b2ef-86a830393919-kube-api-access-kjb69" (OuterVolumeSpecName: "kube-api-access-kjb69") pod "6a404d1a-5e27-40b6-b2ef-86a830393919" (UID: "6a404d1a-5e27-40b6-b2ef-86a830393919"). InnerVolumeSpecName "kube-api-access-kjb69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:28:10.073911 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.073885    2574 scope.go:117] "RemoveContainer" containerID="240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3"
Sep 17 16:28:10.074247 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:28:10.074223    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3\": container with ID starting with 240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3 not found: ID does not exist" containerID="240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3"
Sep 17 16:28:10.074311 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.074256    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3"} err="failed to get container status \"240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3\": rpc error: code = NotFound desc = could not find container \"240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3\": container with ID starting with 240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3 not found: ID does not exist"
Sep 17 16:28:10.074311 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.074275    2574 scope.go:117] "RemoveContainer" containerID="338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48"
Sep 17 16:28:10.074603 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:28:10.074583    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48\": container with ID starting with 338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48 not found: ID does not exist" containerID="338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48"
Sep 17 16:28:10.074683 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.074613    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48"} err="failed to get container status \"338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48\": rpc error: code = NotFound desc = could not find container \"338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48\": container with ID starting with 338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48 not found: ID does not exist"
Sep 17 16:28:10.074683 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.074633    2574 scope.go:117] "RemoveContainer" containerID="051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f"
Sep 17 16:28:10.074922 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:28:10.074904    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f\": container with ID starting with 051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f not found: ID does not exist" containerID="051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f"
Sep 17 16:28:10.075003 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.074930    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f"} err="failed to get container status \"051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f\": rpc error: code = NotFound desc = could not find container \"051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f\": container with ID starting with 051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f not found: ID does not exist"
Sep 17 16:28:10.075003 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.074949    2574 scope.go:117] "RemoveContainer" containerID="240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3"
Sep 17 16:28:10.075243 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.075218    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3"} err="failed to get container status \"240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3\": rpc error: code = NotFound desc = could not find container \"240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3\": container with ID starting with 240b29e5725f344b9bcf9e53fdd019f8bdd234b47a5a32c6cfd0e3396fcf1ee3 not found: ID does not exist"
Sep 17 16:28:10.075314 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.075244    2574 scope.go:117] "RemoveContainer" containerID="338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48"
Sep 17 16:28:10.075554 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.075524    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48"} err="failed to get container status \"338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48\": rpc error: code = NotFound desc = could not find container \"338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48\": container with ID starting with 338de09ffb493f0f241ce548e2f0522f717f30de1eef50fca8817ac7b6910c48 not found: ID does not exist"
Sep 17 16:28:10.075554 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.075545    2574 scope.go:117] "RemoveContainer" containerID="051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f"
Sep 17 16:28:10.075805 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.075779    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f"} err="failed to get container status \"051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f\": rpc error: code = NotFound desc = could not find container \"051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f\": container with ID starting with 051989eb0811c13f27936978e3c7a31183a37ee99795f0695b60f5ba4057c42f not found: ID does not exist"
Sep 17 16:28:10.078579 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.078553    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a404d1a-5e27-40b6-b2ef-86a830393919" (UID: "6a404d1a-5e27-40b6-b2ef-86a830393919"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:28:10.085643 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.085622    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a404d1a-5e27-40b6-b2ef-86a830393919" (UID: "6a404d1a-5e27-40b6-b2ef-86a830393919"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:28:10.160857 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.160830    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a404d1a-5e27-40b6-b2ef-86a830393919-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:28:10.160857 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.160854    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:28:10.160997 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.160865    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjb69\" (UniqueName: \"kubernetes.io/projected/6a404d1a-5e27-40b6-b2ef-86a830393919-kube-api-access-kjb69\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:28:10.160997 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.160876    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a404d1a-5e27-40b6-b2ef-86a830393919-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:28:10.160997 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.160886    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:28:10.160997 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.160895    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:28:10.160997 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.160901    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:28:10.160997 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.160909    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a404d1a-5e27-40b6-b2ef-86a830393919-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:28:10.366538 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.366506    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"]
Sep 17 16:28:10.373155 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:10.373131    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7b6fd469b8vs8ns"]
Sep 17 16:28:11.048394 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:11.048359    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" path="/var/lib/kubelet/pods/6a404d1a-5e27-40b6-b2ef-86a830393919/volumes"
Sep 17 16:28:11.991290 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:11.991265    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"
Sep 17 16:28:12.053751 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.053667    2574 generic.go:356] "Generic (PLEG): container finished" podID="85b116df-f80b-4753-91db-ac4439925715" containerID="6abd63b5dd7061b9e98d5c0099e076e81df030412e676b63514d41eb102d4bdf" exitCode=0
Sep 17 16:28:12.053751 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.053715    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"
Sep 17 16:28:12.053926 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.053748    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz" event={"ID":"85b116df-f80b-4753-91db-ac4439925715","Type":"ContainerDied","Data":"6abd63b5dd7061b9e98d5c0099e076e81df030412e676b63514d41eb102d4bdf"}
Sep 17 16:28:12.053926 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.053797    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz" event={"ID":"85b116df-f80b-4753-91db-ac4439925715","Type":"ContainerDied","Data":"cc363a678c4afcb55acba7e844cb9d89a11a5067b4d1542c95338dc5bf76fd4a"}
Sep 17 16:28:12.053926 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.053829    2574 scope.go:117] "RemoveContainer" containerID="6abd63b5dd7061b9e98d5c0099e076e81df030412e676b63514d41eb102d4bdf"
Sep 17 16:28:12.064591 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.064563    2574 scope.go:117] "RemoveContainer" containerID="6abd63b5dd7061b9e98d5c0099e076e81df030412e676b63514d41eb102d4bdf"
Sep 17 16:28:12.065022 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:28:12.064996    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6abd63b5dd7061b9e98d5c0099e076e81df030412e676b63514d41eb102d4bdf\": container with ID starting with 6abd63b5dd7061b9e98d5c0099e076e81df030412e676b63514d41eb102d4bdf not found: ID does not exist" containerID="6abd63b5dd7061b9e98d5c0099e076e81df030412e676b63514d41eb102d4bdf"
Sep 17 16:28:12.065131 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.065028    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6abd63b5dd7061b9e98d5c0099e076e81df030412e676b63514d41eb102d4bdf"} err="failed to get container status \"6abd63b5dd7061b9e98d5c0099e076e81df030412e676b63514d41eb102d4bdf\": rpc error: code = NotFound desc = could not find container \"6abd63b5dd7061b9e98d5c0099e076e81df030412e676b63514d41eb102d4bdf\": container with ID starting with 6abd63b5dd7061b9e98d5c0099e076e81df030412e676b63514d41eb102d4bdf not found: ID does not exist"
Sep 17 16:28:12.076483 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.076462    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b116df-f80b-4753-91db-ac4439925715-tls-certs\") pod \"85b116df-f80b-4753-91db-ac4439925715\" (UID: \"85b116df-f80b-4753-91db-ac4439925715\") "
Sep 17 16:28:12.076586 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.076525    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8qj6\" (UniqueName: \"kubernetes.io/projected/85b116df-f80b-4753-91db-ac4439925715-kube-api-access-z8qj6\") pod \"85b116df-f80b-4753-91db-ac4439925715\" (UID: \"85b116df-f80b-4753-91db-ac4439925715\") "
Sep 17 16:28:12.078520 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.078498    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b116df-f80b-4753-91db-ac4439925715-kube-api-access-z8qj6" (OuterVolumeSpecName: "kube-api-access-z8qj6") pod "85b116df-f80b-4753-91db-ac4439925715" (UID: "85b116df-f80b-4753-91db-ac4439925715"). InnerVolumeSpecName "kube-api-access-z8qj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:28:12.078616 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.078542    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85b116df-f80b-4753-91db-ac4439925715-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85b116df-f80b-4753-91db-ac4439925715" (UID: "85b116df-f80b-4753-91db-ac4439925715"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:28:12.177217 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.177180    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b116df-f80b-4753-91db-ac4439925715-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:28:12.177217 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.177215    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8qj6\" (UniqueName: \"kubernetes.io/projected/85b116df-f80b-4753-91db-ac4439925715-kube-api-access-z8qj6\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:28:12.376872 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.376841    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"]
Sep 17 16:28:12.380822 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:12.380785    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nfqsz"]
Sep 17 16:28:13.048556 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:13.048520    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85b116df-f80b-4753-91db-ac4439925715" path="/var/lib/kubelet/pods/85b116df-f80b-4753-91db-ac4439925715/volumes"
Sep 17 16:28:16.316424 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:16.316386    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:16.414827 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:16.414790    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:22.697198 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:22.697108    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:24.630999 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.630966    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"]
Sep 17 16:28:24.631564 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631543    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="storage-initializer"
Sep 17 16:28:24.631668 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631567    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="storage-initializer"
Sep 17 16:28:24.631668 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631585    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="storage-initializer"
Sep 17 16:28:24.631668 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631593    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="storage-initializer"
Sep 17 16:28:24.631668 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631603    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main"
Sep 17 16:28:24.631668 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631612    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main"
Sep 17 16:28:24.631668 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631632    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main"
Sep 17 16:28:24.631668 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631640    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main"
Sep 17 16:28:24.631668 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631654    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b116df-f80b-4753-91db-ac4439925715" containerName="main"
Sep 17 16:28:24.631668 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631662    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b116df-f80b-4753-91db-ac4439925715" containerName="main"
Sep 17 16:28:24.632091 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631688    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="llm-d-routing-sidecar"
Sep 17 16:28:24.632091 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631698    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="llm-d-routing-sidecar"
Sep 17 16:28:24.632091 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631792    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68434d9-c908-45f9-8683-9de8ec31ee81" containerName="main"
Sep 17 16:28:24.632091 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631801    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="llm-d-routing-sidecar"
Sep 17 16:28:24.632091 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631811    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a404d1a-5e27-40b6-b2ef-86a830393919" containerName="main"
Sep 17 16:28:24.632091 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.631825    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="85b116df-f80b-4753-91db-ac4439925715" containerName="main"
Sep 17 16:28:24.634986 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.634964    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.637520 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.637496    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-fqvvb\""
Sep 17 16:28:24.637520 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.637514    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 17 16:28:24.637701 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.637519    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 17 16:28:24.637701 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.637505    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:28:24.637701 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.637652    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 17 16:28:24.637953 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.637939    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 17 16:28:24.643691 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.643665    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:24.648402 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.648378    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"]
Sep 17 16:28:24.680760 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.680732    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-model-cache\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.680894 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.680778    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-home\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.680894 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.680804    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.680894 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.680869    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-tmp-dir\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.681011 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.680909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b55a706-1f49-4a05-b131-d23eab75474d-tls-certs\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.681011 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.680926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-dshm\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.681011 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.680941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b55a706-1f49-4a05-b131-d23eab75474d-cabundle-cert\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.681011 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.680985    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b275m\" (UniqueName: \"kubernetes.io/projected/7b55a706-1f49-4a05-b131-d23eab75474d-kube-api-access-b275m\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.781512 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.781469    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-model-cache\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.781708 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.781551    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-home\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.781708 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.781577    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.781708 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.781615    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-tmp-dir\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.781708 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.781687    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b55a706-1f49-4a05-b131-d23eab75474d-tls-certs\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.781939 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.781903    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-model-cache\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.781939 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.781842    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-dshm\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.782049 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.781960    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-tmp-dir\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.782049 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.781964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b55a706-1f49-4a05-b131-d23eab75474d-cabundle-cert\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.782049 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.782013    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.782217 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.782063    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b275m\" (UniqueName: \"kubernetes.io/projected/7b55a706-1f49-4a05-b131-d23eab75474d-kube-api-access-b275m\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.782217 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.782128    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-home\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.782708 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.782674    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b55a706-1f49-4a05-b131-d23eab75474d-cabundle-cert\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.784219 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.784199    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-dshm\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.784508 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.784489    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b55a706-1f49-4a05-b131-d23eab75474d-tls-certs\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.793431 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.793408    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b275m\" (UniqueName: \"kubernetes.io/projected/7b55a706-1f49-4a05-b131-d23eab75474d-kube-api-access-b275m\") pod \"lora-single-adapter-test-kserve-54b6945b84-9d7q9\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:24.947713 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:24.947610    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:25.226719 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:25.226690    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"]
Sep 17 16:28:25.228967 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:28:25.228938    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b55a706_1f49_4a05_b131_d23eab75474d.slice/crio-dc8f78e8364551859cbdf04c6861337ba93df5bdd017119a7e852c36d9198767 WatchSource:0}: Error finding container dc8f78e8364551859cbdf04c6861337ba93df5bdd017119a7e852c36d9198767: Status 404 returned error can't find the container with id dc8f78e8364551859cbdf04c6861337ba93df5bdd017119a7e852c36d9198767
Sep 17 16:28:26.121734 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:26.121688    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" event={"ID":"7b55a706-1f49-4a05-b131-d23eab75474d","Type":"ContainerStarted","Data":"3e8210c489e888c8fd80aad5e4d0b6d23517d9c08cfb3caec911e779843876e9"}
Sep 17 16:28:26.122239 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:26.121767    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" event={"ID":"7b55a706-1f49-4a05-b131-d23eab75474d","Type":"ContainerStarted","Data":"dc8f78e8364551859cbdf04c6861337ba93df5bdd017119a7e852c36d9198767"}
Sep 17 16:28:26.739856 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:26.739821    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:26.839869 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:26.839837    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:28.888594 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:28.888564    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:28.968114 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:28.968076    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:29.139815 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:29.139729    2574 generic.go:356] "Generic (PLEG): container finished" podID="7b55a706-1f49-4a05-b131-d23eab75474d" containerID="3e8210c489e888c8fd80aad5e4d0b6d23517d9c08cfb3caec911e779843876e9" exitCode=0
Sep 17 16:28:29.139815 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:29.139801    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" event={"ID":"7b55a706-1f49-4a05-b131-d23eab75474d","Type":"ContainerDied","Data":"3e8210c489e888c8fd80aad5e4d0b6d23517d9c08cfb3caec911e779843876e9"}
Sep 17 16:28:30.146411 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:30.146371    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" event={"ID":"7b55a706-1f49-4a05-b131-d23eab75474d","Type":"ContainerStarted","Data":"92be2b5136591132982a86e328381543ac5b22ba430ebba61cac04ffbbe41a1b"}
Sep 17 16:28:30.171177 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:30.171124    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" podStartSLOduration=3.291715414 podStartE2EDuration="6.171110149s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.879394735s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:28:30.169024759 +0000 UTC m=+2975.640864604" watchObservedRunningTime="2026-09-17 16:28:30.171110149 +0000 UTC m=+2975.642949994"
Sep 17 16:28:30.948021 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:30.947981    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:32.924378 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:32.924339    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:34.948755 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:34.948656    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:28:34.950286 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:34.950254    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 16:28:44.202954 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:44.202914    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:44.302385 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:44.302347    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:44.948200 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:44.948153    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 16:28:47.756513 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.756478    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"]
Sep 17 16:28:47.759936 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.759912    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:47.762413 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.762374    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:28:47.762972 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.762945    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 16:28:47.763148 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.762997    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 16:28:47.763259 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.762997    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-pwwxw\""
Sep 17 16:28:47.763259 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.763047    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 16:28:47.763366 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.763072    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 16:28:47.778214 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.778183    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"]
Sep 17 16:28:47.795857 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.795814    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:47.899143 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.899090    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:47.899322 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.899217    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:47.899322 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.899259    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsh9x\" (UniqueName: \"kubernetes.io/projected/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-kube-api-access-vsh9x\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:47.899322 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.899291    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:47.899436 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.899390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:47.899516 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.899479    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:47.899516 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.899507    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:47.899585 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:47.899536    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.000255 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.000218    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.000481 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.000334    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsh9x\" (UniqueName: \"kubernetes.io/projected/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-kube-api-access-vsh9x\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.000481 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.000373    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.000481 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.000430    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.000664 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.000502    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.000664 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.000535    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.000664 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.000577    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.000664 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.000617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.000877 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.000705    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.000877 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.000747    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.000877 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.000805    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.001038 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.000991    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.001296 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.001271    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.003001 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.002972    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.003213 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.003193    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.012637 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.012575    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsh9x\" (UniqueName: \"kubernetes.io/projected/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-kube-api-access-vsh9x\") pod \"custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.023757 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.023712    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:48.071933 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.071894    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:48.363330 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:48.363291    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"]
Sep 17 16:28:48.367136 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:28:48.367101    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80b6f03_0a87_40e3_bfc9_cf129a5cd303.slice/crio-58af1520b47cb6a4ec40851cdc32a4fe93cb8f3f5557232da78e08de6a33842a WatchSource:0}: Error finding container 58af1520b47cb6a4ec40851cdc32a4fe93cb8f3f5557232da78e08de6a33842a: Status 404 returned error can't find the container with id 58af1520b47cb6a4ec40851cdc32a4fe93cb8f3f5557232da78e08de6a33842a
Sep 17 16:28:49.248213 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:49.248172    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" event={"ID":"e80b6f03-0a87-40e3-bfc9-cf129a5cd303","Type":"ContainerStarted","Data":"d1ffa249f15558cb5b908aa56f165224ec715615da2f8ff89f8f4e11691ac7b8"}
Sep 17 16:28:49.248637 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:49.248245    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" event={"ID":"e80b6f03-0a87-40e3-bfc9-cf129a5cd303","Type":"ContainerStarted","Data":"58af1520b47cb6a4ec40851cdc32a4fe93cb8f3f5557232da78e08de6a33842a"}
Sep 17 16:28:51.261197 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:51.261121    2574 generic.go:356] "Generic (PLEG): container finished" podID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerID="d1ffa249f15558cb5b908aa56f165224ec715615da2f8ff89f8f4e11691ac7b8" exitCode=0
Sep 17 16:28:51.261588 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:51.261196    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" event={"ID":"e80b6f03-0a87-40e3-bfc9-cf129a5cd303","Type":"ContainerDied","Data":"d1ffa249f15558cb5b908aa56f165224ec715615da2f8ff89f8f4e11691ac7b8"}
Sep 17 16:28:51.565656 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:51.565547    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:52.268376 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:52.268340    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" event={"ID":"e80b6f03-0a87-40e3-bfc9-cf129a5cd303","Type":"ContainerStarted","Data":"5608355792c6db2c960ac8f2fe4b312fab533da10e461bc5085d9a772183092b"}
Sep 17 16:28:52.293028 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:52.292965    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" podStartSLOduration=2.829898396 podStartE2EDuration="5.292942657s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.463044261s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:28:52.290159805 +0000 UTC m=+2997.761999664" watchObservedRunningTime="2026-09-17 16:28:52.292942657 +0000 UTC m=+2997.764782504"
Sep 17 16:28:54.948706 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:54.948661    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 16:28:57.458881 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:57.458837    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:58.000463 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:58.000399    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:58.072351 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:58.072308    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:58.072571 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:58.072383    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:28:58.073999 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:58.073965    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 16:28:58.100629 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:28:58.100602    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:29:04.948381 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:29:04.948321    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 16:29:08.080805 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:29:08.080744    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 16:29:14.948459 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:29:14.948392    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 16:29:18.072362 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:29:18.072317    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 16:29:24.948537 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:29:24.948485    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 16:29:28.072721 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:29:28.072672    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 16:29:34.948961 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:29:34.948852    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 16:29:38.072858 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:29:38.072801    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 16:29:44.958802 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:29:44.958759    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:29:44.966652 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:29:44.966620    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:29:48.073330 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:29:48.073226    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 16:29:58.072993 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:29:58.072939    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 16:30:08.073215 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:08.073174    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 16:30:15.050517 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:15.050471    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-54b6945b84-9d7q9_7b55a706-1f49-4a05-b131-d23eab75474d/storage-initializer/0.log"
Sep 17 16:30:15.061213 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:15.061189    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-54b6945b84-9d7q9_7b55a706-1f49-4a05-b131-d23eab75474d/main/0.log"
Sep 17 16:30:16.681603 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:16.681566    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"]
Sep 17 16:30:16.699117 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:16.699080    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:30:18.073403 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:18.073344    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 16:30:28.073199 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:28.073156    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 16:30:31.705341 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:31.705271    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" containerName="main" containerID="cri-o://92be2b5136591132982a86e328381543ac5b22ba430ebba61cac04ffbbe41a1b" gracePeriod=45
Sep 17 16:30:31.959892 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:31.959793    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 16:30:32.687063 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.687033    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:30:32.790871 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.790795    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b55a706-1f49-4a05-b131-d23eab75474d-tls-certs\") pod \"7b55a706-1f49-4a05-b131-d23eab75474d\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") "
Sep 17 16:30:32.790871 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.790845    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-dshm\") pod \"7b55a706-1f49-4a05-b131-d23eab75474d\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") "
Sep 17 16:30:32.791388 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.790873    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b275m\" (UniqueName: \"kubernetes.io/projected/7b55a706-1f49-4a05-b131-d23eab75474d-kube-api-access-b275m\") pod \"7b55a706-1f49-4a05-b131-d23eab75474d\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") "
Sep 17 16:30:32.791388 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.790915    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b55a706-1f49-4a05-b131-d23eab75474d-cabundle-cert\") pod \"7b55a706-1f49-4a05-b131-d23eab75474d\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") "
Sep 17 16:30:32.791388 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.790972    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-model-cache\") pod \"7b55a706-1f49-4a05-b131-d23eab75474d\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") "
Sep 17 16:30:32.791388 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.791010    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-kserve-provision-location\") pod \"7b55a706-1f49-4a05-b131-d23eab75474d\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") "
Sep 17 16:30:32.791388 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.791036    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-tmp-dir\") pod \"7b55a706-1f49-4a05-b131-d23eab75474d\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") "
Sep 17 16:30:32.791388 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.791077    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-home\") pod \"7b55a706-1f49-4a05-b131-d23eab75474d\" (UID: \"7b55a706-1f49-4a05-b131-d23eab75474d\") "
Sep 17 16:30:32.791388 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.791253    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b55a706-1f49-4a05-b131-d23eab75474d" (UID: "7b55a706-1f49-4a05-b131-d23eab75474d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:30:32.791388 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.791284    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b55a706-1f49-4a05-b131-d23eab75474d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7b55a706-1f49-4a05-b131-d23eab75474d" (UID: "7b55a706-1f49-4a05-b131-d23eab75474d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:30:32.791855 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.791433    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b55a706-1f49-4a05-b131-d23eab75474d-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:30:32.791855 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.791487    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:30:32.791855 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.791628    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-home" (OuterVolumeSpecName: "home") pod "7b55a706-1f49-4a05-b131-d23eab75474d" (UID: "7b55a706-1f49-4a05-b131-d23eab75474d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:30:32.793278 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.793237    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b55a706-1f49-4a05-b131-d23eab75474d-kube-api-access-b275m" (OuterVolumeSpecName: "kube-api-access-b275m") pod "7b55a706-1f49-4a05-b131-d23eab75474d" (UID: "7b55a706-1f49-4a05-b131-d23eab75474d"). InnerVolumeSpecName "kube-api-access-b275m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:30:32.793807 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.793787    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b55a706-1f49-4a05-b131-d23eab75474d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b55a706-1f49-4a05-b131-d23eab75474d" (UID: "7b55a706-1f49-4a05-b131-d23eab75474d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:30:32.794049 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.794032    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-dshm" (OuterVolumeSpecName: "dshm") pod "7b55a706-1f49-4a05-b131-d23eab75474d" (UID: "7b55a706-1f49-4a05-b131-d23eab75474d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:30:32.807963 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.807931    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b55a706-1f49-4a05-b131-d23eab75474d" (UID: "7b55a706-1f49-4a05-b131-d23eab75474d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:30:32.817108 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.817069    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b55a706-1f49-4a05-b131-d23eab75474d" (UID: "7b55a706-1f49-4a05-b131-d23eab75474d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:30:32.859663 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.859626    2574 generic.go:356] "Generic (PLEG): container finished" podID="7b55a706-1f49-4a05-b131-d23eab75474d" containerID="92be2b5136591132982a86e328381543ac5b22ba430ebba61cac04ffbbe41a1b" exitCode=0
Sep 17 16:30:32.859832 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.859659    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" event={"ID":"7b55a706-1f49-4a05-b131-d23eab75474d","Type":"ContainerDied","Data":"92be2b5136591132982a86e328381543ac5b22ba430ebba61cac04ffbbe41a1b"}
Sep 17 16:30:32.859832 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.859711    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"
Sep 17 16:30:32.859832 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.859740    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9" event={"ID":"7b55a706-1f49-4a05-b131-d23eab75474d","Type":"ContainerDied","Data":"dc8f78e8364551859cbdf04c6861337ba93df5bdd017119a7e852c36d9198767"}
Sep 17 16:30:32.859832 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.859763    2574 scope.go:117] "RemoveContainer" containerID="92be2b5136591132982a86e328381543ac5b22ba430ebba61cac04ffbbe41a1b"
Sep 17 16:30:32.868976 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.868955    2574 scope.go:117] "RemoveContainer" containerID="3e8210c489e888c8fd80aad5e4d0b6d23517d9c08cfb3caec911e779843876e9"
Sep 17 16:30:32.877749 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.877721    2574 scope.go:117] "RemoveContainer" containerID="92be2b5136591132982a86e328381543ac5b22ba430ebba61cac04ffbbe41a1b"
Sep 17 16:30:32.878028 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:30:32.878010    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92be2b5136591132982a86e328381543ac5b22ba430ebba61cac04ffbbe41a1b\": container with ID starting with 92be2b5136591132982a86e328381543ac5b22ba430ebba61cac04ffbbe41a1b not found: ID does not exist" containerID="92be2b5136591132982a86e328381543ac5b22ba430ebba61cac04ffbbe41a1b"
Sep 17 16:30:32.878099 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.878037    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92be2b5136591132982a86e328381543ac5b22ba430ebba61cac04ffbbe41a1b"} err="failed to get container status \"92be2b5136591132982a86e328381543ac5b22ba430ebba61cac04ffbbe41a1b\": rpc error: code = NotFound desc = could not find container \"92be2b5136591132982a86e328381543ac5b22ba430ebba61cac04ffbbe41a1b\": container with ID starting with 92be2b5136591132982a86e328381543ac5b22ba430ebba61cac04ffbbe41a1b not found: ID does not exist"
Sep 17 16:30:32.878099 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.878053    2574 scope.go:117] "RemoveContainer" containerID="3e8210c489e888c8fd80aad5e4d0b6d23517d9c08cfb3caec911e779843876e9"
Sep 17 16:30:32.878306 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:30:32.878289    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8210c489e888c8fd80aad5e4d0b6d23517d9c08cfb3caec911e779843876e9\": container with ID starting with 3e8210c489e888c8fd80aad5e4d0b6d23517d9c08cfb3caec911e779843876e9 not found: ID does not exist" containerID="3e8210c489e888c8fd80aad5e4d0b6d23517d9c08cfb3caec911e779843876e9"
Sep 17 16:30:32.878351 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.878310    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8210c489e888c8fd80aad5e4d0b6d23517d9c08cfb3caec911e779843876e9"} err="failed to get container status \"3e8210c489e888c8fd80aad5e4d0b6d23517d9c08cfb3caec911e779843876e9\": rpc error: code = NotFound desc = could not find container \"3e8210c489e888c8fd80aad5e4d0b6d23517d9c08cfb3caec911e779843876e9\": container with ID starting with 3e8210c489e888c8fd80aad5e4d0b6d23517d9c08cfb3caec911e779843876e9 not found: ID does not exist"
Sep 17 16:30:32.886994 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.886967    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"]
Sep 17 16:30:32.889341 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.889320    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54b6945b84-9d7q9"]
Sep 17 16:30:32.892650 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.892627    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b55a706-1f49-4a05-b131-d23eab75474d-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:30:32.892742 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.892652    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:30:32.892742 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.892666    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b275m\" (UniqueName: \"kubernetes.io/projected/7b55a706-1f49-4a05-b131-d23eab75474d-kube-api-access-b275m\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:30:32.892742 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.892680    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:30:32.892742 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.892694    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:30:32.892742 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:32.892705    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b55a706-1f49-4a05-b131-d23eab75474d-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:30:33.048879 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:33.048787    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" path="/var/lib/kubelet/pods/7b55a706-1f49-4a05-b131-d23eab75474d/volumes"
Sep 17 16:30:36.848304 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:36.848265    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:30:36.950035 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:36.949988    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:30:37.049134 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:37.049098    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:30:38.072558 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:38.072509    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 16:30:43.279566 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:43.279521    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:30:43.379890 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:43.379835    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:30:44.006847 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.006812    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"]
Sep 17 16:30:44.007280 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.007260    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" containerName="storage-initializer"
Sep 17 16:30:44.007376 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.007282    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" containerName="storage-initializer"
Sep 17 16:30:44.007376 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.007302    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" containerName="main"
Sep 17 16:30:44.007376 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.007310    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" containerName="main"
Sep 17 16:30:44.007664 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.007416    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b55a706-1f49-4a05-b131-d23eab75474d" containerName="main"
Sep 17 16:30:44.013467 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.013419    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.018575 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.018513    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 17 16:30:44.018737 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.018605    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 17 16:30:44.018737 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.018629    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-4bxdx\""
Sep 17 16:30:44.018868 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.018844    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 17 16:30:44.019178 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.019160    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:30:44.019896 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.019872    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 17 16:30:44.030022 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.029998    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"]
Sep 17 16:30:44.041069 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.041036    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:30:44.097576 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.097538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.097758 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.097586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqmmf\" (UniqueName: \"kubernetes.io/projected/2bb45650-9553-4f9b-bae1-ec9d4231c294-kube-api-access-rqmmf\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.097758 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.097649    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb45650-9553-4f9b-bae1-ec9d4231c294-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.097758 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.097715    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.097918 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.097781    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bb45650-9553-4f9b-bae1-ec9d4231c294-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.097918 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.097799    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-model-cache\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.097918 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.097870    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-home\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.097918 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.097910    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-dshm\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.199285 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.199240    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.199518 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.199310    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bb45650-9553-4f9b-bae1-ec9d4231c294-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.199518 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.199359    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-model-cache\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.199518 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.199412    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-home\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.199518 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.199431    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-dshm\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.199518 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.199494    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.199817 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.199538    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqmmf\" (UniqueName: \"kubernetes.io/projected/2bb45650-9553-4f9b-bae1-ec9d4231c294-kube-api-access-rqmmf\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.199817 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.199576    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb45650-9553-4f9b-bae1-ec9d4231c294-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.199817 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.199802    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-model-cache\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.199919 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.199851    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.199919 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.199888    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.200268 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.200242    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-home\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.200367 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.200322    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bb45650-9553-4f9b-bae1-ec9d4231c294-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.202182 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.202148    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-dshm\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.202338 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.202322    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb45650-9553-4f9b-bae1-ec9d4231c294-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.207928 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.207905    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqmmf\" (UniqueName: \"kubernetes.io/projected/2bb45650-9553-4f9b-bae1-ec9d4231c294-kube-api-access-rqmmf\") pod \"lora-multiple-adapters-test-kserve-6c97b497f9-zftql\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.324562 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.324476    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:44.614369 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.614333    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"]
Sep 17 16:30:44.616262 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:30:44.616225    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb45650_9553_4f9b_bae1_ec9d4231c294.slice/crio-c7edc10f36fe6a934748ada6ecfdc310303c960d234b17e934946ca9a1988378 WatchSource:0}: Error finding container c7edc10f36fe6a934748ada6ecfdc310303c960d234b17e934946ca9a1988378: Status 404 returned error can't find the container with id c7edc10f36fe6a934748ada6ecfdc310303c960d234b17e934946ca9a1988378
Sep 17 16:30:44.928561 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.928458    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" event={"ID":"2bb45650-9553-4f9b-bae1-ec9d4231c294","Type":"ContainerStarted","Data":"84d8e80a80b000bbf6ce77868fdc02691cc5f647ef06c02754e8ae1a945fd50c"}
Sep 17 16:30:44.928561 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:44.928528    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" event={"ID":"2bb45650-9553-4f9b-bae1-ec9d4231c294","Type":"ContainerStarted","Data":"c7edc10f36fe6a934748ada6ecfdc310303c960d234b17e934946ca9a1988378"}
Sep 17 16:30:47.953200 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:47.953149    2574 generic.go:356] "Generic (PLEG): container finished" podID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerID="84d8e80a80b000bbf6ce77868fdc02691cc5f647ef06c02754e8ae1a945fd50c" exitCode=0
Sep 17 16:30:47.953692 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:47.953234    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" event={"ID":"2bb45650-9553-4f9b-bae1-ec9d4231c294","Type":"ContainerDied","Data":"84d8e80a80b000bbf6ce77868fdc02691cc5f647ef06c02754e8ae1a945fd50c"}
Sep 17 16:30:48.072411 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:48.072355    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 16:30:48.961802 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:48.961758    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" event={"ID":"2bb45650-9553-4f9b-bae1-ec9d4231c294","Type":"ContainerStarted","Data":"9a4bdfd391a1f7e7d056226226e430a9e45acc5ded771c27616460a57805739b"}
Sep 17 16:30:48.987880 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:48.987808    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" podStartSLOduration=2.7555805270000002 podStartE2EDuration="5.98778452s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.232203993s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:30:48.984903783 +0000 UTC m=+3114.456743621" watchObservedRunningTime="2026-09-17 16:30:48.98778452 +0000 UTC m=+3114.459624367"
Sep 17 16:30:49.324878 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:49.324792    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:51.781478 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:51.781427    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:30:54.324750 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:54.324718    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:30:54.326165 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:54.326135    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" podUID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 16:30:58.082666 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:58.082633    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:30:58.090538 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:30:58.090514    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:31:04.325118 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:04.325075    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" podUID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 16:31:07.973653 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:07.973614    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"]
Sep 17 16:31:09.224274 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:09.224236    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:31:09.309145 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:09.309105    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:31:14.325019 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:14.324926    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" podUID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 16:31:22.996918 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:22.996782    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main" containerID="cri-o://5608355792c6db2c960ac8f2fe4b312fab533da10e461bc5085d9a772183092b" gracePeriod=45
Sep 17 16:31:23.741417 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.741391    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:31:23.872301 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.872266    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-model-cache\") pod \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") "
Sep 17 16:31:23.872522 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.872320    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-kserve-provision-location\") pod \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") "
Sep 17 16:31:23.872522 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.872358    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-dshm\") pod \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") "
Sep 17 16:31:23.872522 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.872418    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-tmp-dir\") pod \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") "
Sep 17 16:31:23.872522 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.872479    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-tls-certs\") pod \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") "
Sep 17 16:31:23.872522 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.872506    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-home\") pod \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") "
Sep 17 16:31:23.872790 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.872553    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsh9x\" (UniqueName: \"kubernetes.io/projected/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-kube-api-access-vsh9x\") pod \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") "
Sep 17 16:31:23.872790 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.872596    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-cabundle-cert\") pod \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\" (UID: \"e80b6f03-0a87-40e3-bfc9-cf129a5cd303\") "
Sep 17 16:31:23.872790 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.872632    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-model-cache" (OuterVolumeSpecName: "model-cache") pod "e80b6f03-0a87-40e3-bfc9-cf129a5cd303" (UID: "e80b6f03-0a87-40e3-bfc9-cf129a5cd303"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:31:23.872935 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.872888    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:31:23.873108 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.873078    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e80b6f03-0a87-40e3-bfc9-cf129a5cd303" (UID: "e80b6f03-0a87-40e3-bfc9-cf129a5cd303"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:31:23.873427 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.873389    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-home" (OuterVolumeSpecName: "home") pod "e80b6f03-0a87-40e3-bfc9-cf129a5cd303" (UID: "e80b6f03-0a87-40e3-bfc9-cf129a5cd303"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:31:23.875047 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.875011    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-kube-api-access-vsh9x" (OuterVolumeSpecName: "kube-api-access-vsh9x") pod "e80b6f03-0a87-40e3-bfc9-cf129a5cd303" (UID: "e80b6f03-0a87-40e3-bfc9-cf129a5cd303"). InnerVolumeSpecName "kube-api-access-vsh9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:31:23.875155 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.875079    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e80b6f03-0a87-40e3-bfc9-cf129a5cd303" (UID: "e80b6f03-0a87-40e3-bfc9-cf129a5cd303"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:31:23.875217 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.875185    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-dshm" (OuterVolumeSpecName: "dshm") pod "e80b6f03-0a87-40e3-bfc9-cf129a5cd303" (UID: "e80b6f03-0a87-40e3-bfc9-cf129a5cd303"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:31:23.887753 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.887726    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e80b6f03-0a87-40e3-bfc9-cf129a5cd303" (UID: "e80b6f03-0a87-40e3-bfc9-cf129a5cd303"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:31:23.896097 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.896068    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e80b6f03-0a87-40e3-bfc9-cf129a5cd303" (UID: "e80b6f03-0a87-40e3-bfc9-cf129a5cd303"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:31:23.973381 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.973340    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:31:23.973568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.973398    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:31:23.973568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.973407    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:31:23.973568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.973415    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:31:23.973568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.973425    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsh9x\" (UniqueName: \"kubernetes.io/projected/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-kube-api-access-vsh9x\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:31:23.973568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.973436    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:31:23.973568 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:23.973469    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80b6f03-0a87-40e3-bfc9-cf129a5cd303-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:31:24.145305 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:24.145223    2574 generic.go:356] "Generic (PLEG): container finished" podID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerID="5608355792c6db2c960ac8f2fe4b312fab533da10e461bc5085d9a772183092b" exitCode=0
Sep 17 16:31:24.145305 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:24.145264    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" event={"ID":"e80b6f03-0a87-40e3-bfc9-cf129a5cd303","Type":"ContainerDied","Data":"5608355792c6db2c960ac8f2fe4b312fab533da10e461bc5085d9a772183092b"}
Sep 17 16:31:24.145305 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:24.145295    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"
Sep 17 16:31:24.145305 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:24.145314    2574 scope.go:117] "RemoveContainer" containerID="5608355792c6db2c960ac8f2fe4b312fab533da10e461bc5085d9a772183092b"
Sep 17 16:31:24.145884 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:24.145303    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx" event={"ID":"e80b6f03-0a87-40e3-bfc9-cf129a5cd303","Type":"ContainerDied","Data":"58af1520b47cb6a4ec40851cdc32a4fe93cb8f3f5557232da78e08de6a33842a"}
Sep 17 16:31:24.155017 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:24.155002    2574 scope.go:117] "RemoveContainer" containerID="d1ffa249f15558cb5b908aa56f165224ec715615da2f8ff89f8f4e11691ac7b8"
Sep 17 16:31:24.163056 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:24.163038    2574 scope.go:117] "RemoveContainer" containerID="5608355792c6db2c960ac8f2fe4b312fab533da10e461bc5085d9a772183092b"
Sep 17 16:31:24.163324 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:31:24.163298    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5608355792c6db2c960ac8f2fe4b312fab533da10e461bc5085d9a772183092b\": container with ID starting with 5608355792c6db2c960ac8f2fe4b312fab533da10e461bc5085d9a772183092b not found: ID does not exist" containerID="5608355792c6db2c960ac8f2fe4b312fab533da10e461bc5085d9a772183092b"
Sep 17 16:31:24.163370 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:24.163338    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5608355792c6db2c960ac8f2fe4b312fab533da10e461bc5085d9a772183092b"} err="failed to get container status \"5608355792c6db2c960ac8f2fe4b312fab533da10e461bc5085d9a772183092b\": rpc error: code = NotFound desc = could not find container \"5608355792c6db2c960ac8f2fe4b312fab533da10e461bc5085d9a772183092b\": container with ID starting with 5608355792c6db2c960ac8f2fe4b312fab533da10e461bc5085d9a772183092b not found: ID does not exist"
Sep 17 16:31:24.163370 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:24.163363    2574 scope.go:117] "RemoveContainer" containerID="d1ffa249f15558cb5b908aa56f165224ec715615da2f8ff89f8f4e11691ac7b8"
Sep 17 16:31:24.163633 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:31:24.163613    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ffa249f15558cb5b908aa56f165224ec715615da2f8ff89f8f4e11691ac7b8\": container with ID starting with d1ffa249f15558cb5b908aa56f165224ec715615da2f8ff89f8f4e11691ac7b8 not found: ID does not exist" containerID="d1ffa249f15558cb5b908aa56f165224ec715615da2f8ff89f8f4e11691ac7b8"
Sep 17 16:31:24.163698 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:24.163643    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ffa249f15558cb5b908aa56f165224ec715615da2f8ff89f8f4e11691ac7b8"} err="failed to get container status \"d1ffa249f15558cb5b908aa56f165224ec715615da2f8ff89f8f4e11691ac7b8\": rpc error: code = NotFound desc = could not find container \"d1ffa249f15558cb5b908aa56f165224ec715615da2f8ff89f8f4e11691ac7b8\": container with ID starting with d1ffa249f15558cb5b908aa56f165224ec715615da2f8ff89f8f4e11691ac7b8 not found: ID does not exist"
Sep 17 16:31:24.170712 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:24.170685    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"]
Sep 17 16:31:24.174044 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:24.174017    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-57fb8b667-whkqx"]
Sep 17 16:31:24.324914 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:24.324879    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" podUID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 16:31:25.048774 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:25.048734    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" path="/var/lib/kubelet/pods/e80b6f03-0a87-40e3-bfc9-cf129a5cd303/volumes"
Sep 17 16:31:34.325714 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:34.325665    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" podUID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 16:31:44.325698 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:44.325648    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" podUID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 16:31:54.325773 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:54.325732    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" podUID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 16:31:58.095657 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:58.095615    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:31:58.192122 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:58.192086    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:31:58.440058 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:58.440005    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:31:58.557781 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:31:58.557739    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:32:01.611069 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:01.611023    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:32:04.335260 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:04.335231    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:32:04.343431 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:04.343405    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:32:05.970258 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:05.970219    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:32:07.599675 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.599637    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"]
Sep 17 16:32:07.600042 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.600024    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main"
Sep 17 16:32:07.600042 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.600038    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main"
Sep 17 16:32:07.600118 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.600052    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="storage-initializer"
Sep 17 16:32:07.600118 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.600058    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="storage-initializer"
Sep 17 16:32:07.600182 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.600126    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e80b6f03-0a87-40e3-bfc9-cf129a5cd303" containerName="main"
Sep 17 16:32:07.603494 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.603471    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.605880 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.605855    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:32:07.606054 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.605939    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 16:32:07.606054 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.605944    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 16:32:07.606054 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.605971    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-m72m4\""
Sep 17 16:32:07.606292 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.606273    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 16:32:07.606406 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.606387    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 16:32:07.616817 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.616792    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"]
Sep 17 16:32:07.618625 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.618597    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:32:07.671434 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.671402    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.671632 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.671492    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.671632 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.671522    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.671632 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.671551    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h6mm\" (UniqueName: \"kubernetes.io/projected/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-kube-api-access-6h6mm\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.671632 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.671586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.671780 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.671639    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.671780 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.671669    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-home\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.671780 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.671694    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.772146 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.772104    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h6mm\" (UniqueName: \"kubernetes.io/projected/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-kube-api-access-6h6mm\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.772146 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.772151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.772411 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.772190    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.772411 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.772211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-home\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.772411 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.772232    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.772411 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.772342    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.772411 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.772400    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.772689 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.772433    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.772750 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.772702    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.772809 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.772787    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.772898 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.772874    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-home\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.772998 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.772974    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.773077 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.773053    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.774781 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.774753    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.775173 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.775153    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.781846 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.781817    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h6mm\" (UniqueName: \"kubernetes.io/projected/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-kube-api-access-6h6mm\") pod \"router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.851562 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.851529    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"]
Sep 17 16:32:07.856257 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.856232    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"
Sep 17 16:32:07.858598 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.858579    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-qgh8m\""
Sep 17 16:32:07.864479 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.864426    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"]
Sep 17 16:32:07.915030 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.914998    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:07.974082 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.973988    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9xhw\" (UniqueName: \"kubernetes.io/projected/e329d53d-0466-43f0-a443-654acdb32384-kube-api-access-p9xhw\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m\" (UID: \"e329d53d-0466-43f0-a443-654acdb32384\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"
Sep 17 16:32:07.974082 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:07.974064    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e329d53d-0466-43f0-a443-654acdb32384-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m\" (UID: \"e329d53d-0466-43f0-a443-654acdb32384\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"
Sep 17 16:32:08.075689 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:08.075643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9xhw\" (UniqueName: \"kubernetes.io/projected/e329d53d-0466-43f0-a443-654acdb32384-kube-api-access-p9xhw\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m\" (UID: \"e329d53d-0466-43f0-a443-654acdb32384\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"
Sep 17 16:32:08.075881 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:08.075746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e329d53d-0466-43f0-a443-654acdb32384-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m\" (UID: \"e329d53d-0466-43f0-a443-654acdb32384\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"
Sep 17 16:32:08.078255 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:08.078222    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e329d53d-0466-43f0-a443-654acdb32384-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m\" (UID: \"e329d53d-0466-43f0-a443-654acdb32384\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"
Sep 17 16:32:08.085139 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:08.085113    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9xhw\" (UniqueName: \"kubernetes.io/projected/e329d53d-0466-43f0-a443-654acdb32384-kube-api-access-p9xhw\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m\" (UID: \"e329d53d-0466-43f0-a443-654acdb32384\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"
Sep 17 16:32:08.159538 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:08.159417    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:32:08.168332 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:08.168299    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"
Sep 17 16:32:08.192106 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:08.191841    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"]
Sep 17 16:32:08.194279 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:32:08.194253    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0ea49d_2dcf_46e1_9df8_b9154fd71b20.slice/crio-098aa089d792b832631b96d6e967800a003f3502a10b7b38a2f204d492f36143 WatchSource:0}: Error finding container 098aa089d792b832631b96d6e967800a003f3502a10b7b38a2f204d492f36143: Status 404 returned error can't find the container with id 098aa089d792b832631b96d6e967800a003f3502a10b7b38a2f204d492f36143
Sep 17 16:32:08.217898 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:08.217865    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:32:08.375957 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:08.375923    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" event={"ID":"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20","Type":"ContainerStarted","Data":"6e0660003ca33c332db51c9501e45d398f303bad6a623d4e93effe2b2e431eb1"}
Sep 17 16:32:08.376133 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:08.375980    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" event={"ID":"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20","Type":"ContainerStarted","Data":"098aa089d792b832631b96d6e967800a003f3502a10b7b38a2f204d492f36143"}
Sep 17 16:32:08.447875 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:08.447850    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"]
Sep 17 16:32:08.450007 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:32:08.449979    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode329d53d_0466_43f0_a443_654acdb32384.slice/crio-1f9e824816010c4036509000a648d57518cfafc57470b4174d1c9e041e5ec11f WatchSource:0}: Error finding container 1f9e824816010c4036509000a648d57518cfafc57470b4174d1c9e041e5ec11f: Status 404 returned error can't find the container with id 1f9e824816010c4036509000a648d57518cfafc57470b4174d1c9e041e5ec11f
Sep 17 16:32:09.383033 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:09.382996    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m" event={"ID":"e329d53d-0466-43f0-a443-654acdb32384","Type":"ContainerStarted","Data":"0d075ef75defbbbd0bb1fcdc4e81954c0668b1145885ae031c3ec4e5e8fa5ba5"}
Sep 17 16:32:09.383455 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:09.383061    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m" event={"ID":"e329d53d-0466-43f0-a443-654acdb32384","Type":"ContainerStarted","Data":"1f9e824816010c4036509000a648d57518cfafc57470b4174d1c9e041e5ec11f"}
Sep 17 16:32:09.383455 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:09.383301    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"
Sep 17 16:32:09.400898 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:09.400848    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m" podStartSLOduration=2.400834519 podStartE2EDuration="2.400834519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:32:09.398882252 +0000 UTC m=+3194.870722099" watchObservedRunningTime="2026-09-17 16:32:09.400834519 +0000 UTC m=+3194.872674392"
Sep 17 16:32:11.394809 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:11.394773    2574 generic.go:356] "Generic (PLEG): container finished" podID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerID="6e0660003ca33c332db51c9501e45d398f303bad6a623d4e93effe2b2e431eb1" exitCode=0
Sep 17 16:32:11.395164 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:11.394844    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" event={"ID":"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20","Type":"ContainerDied","Data":"6e0660003ca33c332db51c9501e45d398f303bad6a623d4e93effe2b2e431eb1"}
Sep 17 16:32:11.941787 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:11.941751    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:32:12.041546 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:12.041506    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:32:12.401783 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:12.401752    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" event={"ID":"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20","Type":"ContainerStarted","Data":"d702811dd14340784a0aa0c34aed4973de5ac89523d02bff54b98e396f307f34"}
Sep 17 16:32:12.425828 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:12.425730    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" podStartSLOduration=3.089902915 podStartE2EDuration="5.425708283s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.335805368s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:32:12.424758114 +0000 UTC m=+3197.896597984" watchObservedRunningTime="2026-09-17 16:32:12.425708283 +0000 UTC m=+3197.897548129"
Sep 17 16:32:17.915965 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:17.915920    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:17.916571 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:17.915996    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:32:17.917702 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:17.917677    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 16:32:27.916122 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:27.916059    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 16:32:37.916285 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:37.916229    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 16:32:40.390875 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:40.390837    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"
Sep 17 16:32:47.915925 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:47.915825    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 16:32:54.484881 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:54.484842    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6c97b497f9-zftql_2bb45650-9553-4f9b-bae1-ec9d4231c294/storage-initializer/0.log"
Sep 17 16:32:54.496381 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:54.496352    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6c97b497f9-zftql_2bb45650-9553-4f9b-bae1-ec9d4231c294/main/0.log"
Sep 17 16:32:55.487786 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:55.487751    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"]
Sep 17 16:32:55.506210 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:55.506169    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:32:57.915384 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:32:57.915340    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 16:33:07.915813 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:07.915762    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 16:33:10.512961 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:10.512916    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" podUID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerName="main" containerID="cri-o://9a4bdfd391a1f7e7d056226226e430a9e45acc5ded771c27616460a57805739b" gracePeriod=45
Sep 17 16:33:11.192677 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.192648    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:33:11.268800 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.268725    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-home\") pod \"2bb45650-9553-4f9b-bae1-ec9d4231c294\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") "
Sep 17 16:33:11.268800 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.268762    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-model-cache\") pod \"2bb45650-9553-4f9b-bae1-ec9d4231c294\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") "
Sep 17 16:33:11.268800 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.268795    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-kserve-provision-location\") pod \"2bb45650-9553-4f9b-bae1-ec9d4231c294\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") "
Sep 17 16:33:11.269071 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.268819    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-tmp-dir\") pod \"2bb45650-9553-4f9b-bae1-ec9d4231c294\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") "
Sep 17 16:33:11.269071 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.268862    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bb45650-9553-4f9b-bae1-ec9d4231c294-cabundle-cert\") pod \"2bb45650-9553-4f9b-bae1-ec9d4231c294\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") "
Sep 17 16:33:11.269071 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.268896    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-dshm\") pod \"2bb45650-9553-4f9b-bae1-ec9d4231c294\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") "
Sep 17 16:33:11.269071 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.268914    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqmmf\" (UniqueName: \"kubernetes.io/projected/2bb45650-9553-4f9b-bae1-ec9d4231c294-kube-api-access-rqmmf\") pod \"2bb45650-9553-4f9b-bae1-ec9d4231c294\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") "
Sep 17 16:33:11.269071 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.268930    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb45650-9553-4f9b-bae1-ec9d4231c294-tls-certs\") pod \"2bb45650-9553-4f9b-bae1-ec9d4231c294\" (UID: \"2bb45650-9553-4f9b-bae1-ec9d4231c294\") "
Sep 17 16:33:11.269358 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.269091    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-model-cache" (OuterVolumeSpecName: "model-cache") pod "2bb45650-9553-4f9b-bae1-ec9d4231c294" (UID: "2bb45650-9553-4f9b-bae1-ec9d4231c294"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:33:11.269358 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.269200    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:33:11.269536 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.269508    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-home" (OuterVolumeSpecName: "home") pod "2bb45650-9553-4f9b-bae1-ec9d4231c294" (UID: "2bb45650-9553-4f9b-bae1-ec9d4231c294"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:33:11.269856 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.269831    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb45650-9553-4f9b-bae1-ec9d4231c294-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2bb45650-9553-4f9b-bae1-ec9d4231c294" (UID: "2bb45650-9553-4f9b-bae1-ec9d4231c294"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:33:11.271833 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.271797    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-dshm" (OuterVolumeSpecName: "dshm") pod "2bb45650-9553-4f9b-bae1-ec9d4231c294" (UID: "2bb45650-9553-4f9b-bae1-ec9d4231c294"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:33:11.271833 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.271799    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb45650-9553-4f9b-bae1-ec9d4231c294-kube-api-access-rqmmf" (OuterVolumeSpecName: "kube-api-access-rqmmf") pod "2bb45650-9553-4f9b-bae1-ec9d4231c294" (UID: "2bb45650-9553-4f9b-bae1-ec9d4231c294"). InnerVolumeSpecName "kube-api-access-rqmmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:33:11.271996 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.271973    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb45650-9553-4f9b-bae1-ec9d4231c294-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2bb45650-9553-4f9b-bae1-ec9d4231c294" (UID: "2bb45650-9553-4f9b-bae1-ec9d4231c294"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:33:11.280029 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.280000    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2bb45650-9553-4f9b-bae1-ec9d4231c294" (UID: "2bb45650-9553-4f9b-bae1-ec9d4231c294"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:33:11.300176 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.300147    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2bb45650-9553-4f9b-bae1-ec9d4231c294" (UID: "2bb45650-9553-4f9b-bae1-ec9d4231c294"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:33:11.370606 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.370561    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqmmf\" (UniqueName: \"kubernetes.io/projected/2bb45650-9553-4f9b-bae1-ec9d4231c294-kube-api-access-rqmmf\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:33:11.370606 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.370595    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb45650-9553-4f9b-bae1-ec9d4231c294-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:33:11.370606 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.370611    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:33:11.370852 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.370651    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:33:11.370852 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.370664    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:33:11.370852 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.370676    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bb45650-9553-4f9b-bae1-ec9d4231c294-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:33:11.370852 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.370688    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb45650-9553-4f9b-bae1-ec9d4231c294-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:33:11.731386 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.731346    2574 generic.go:356] "Generic (PLEG): container finished" podID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerID="9a4bdfd391a1f7e7d056226226e430a9e45acc5ded771c27616460a57805739b" exitCode=0
Sep 17 16:33:11.731888 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.731397    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"
Sep 17 16:33:11.731888 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.731431    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" event={"ID":"2bb45650-9553-4f9b-bae1-ec9d4231c294","Type":"ContainerDied","Data":"9a4bdfd391a1f7e7d056226226e430a9e45acc5ded771c27616460a57805739b"}
Sep 17 16:33:11.731888 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.731497    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql" event={"ID":"2bb45650-9553-4f9b-bae1-ec9d4231c294","Type":"ContainerDied","Data":"c7edc10f36fe6a934748ada6ecfdc310303c960d234b17e934946ca9a1988378"}
Sep 17 16:33:11.731888 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.731521    2574 scope.go:117] "RemoveContainer" containerID="9a4bdfd391a1f7e7d056226226e430a9e45acc5ded771c27616460a57805739b"
Sep 17 16:33:11.742404 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.742386    2574 scope.go:117] "RemoveContainer" containerID="84d8e80a80b000bbf6ce77868fdc02691cc5f647ef06c02754e8ae1a945fd50c"
Sep 17 16:33:11.751112 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.751093    2574 scope.go:117] "RemoveContainer" containerID="9a4bdfd391a1f7e7d056226226e430a9e45acc5ded771c27616460a57805739b"
Sep 17 16:33:11.751414 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:33:11.751395    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4bdfd391a1f7e7d056226226e430a9e45acc5ded771c27616460a57805739b\": container with ID starting with 9a4bdfd391a1f7e7d056226226e430a9e45acc5ded771c27616460a57805739b not found: ID does not exist" containerID="9a4bdfd391a1f7e7d056226226e430a9e45acc5ded771c27616460a57805739b"
Sep 17 16:33:11.751512 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.751420    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4bdfd391a1f7e7d056226226e430a9e45acc5ded771c27616460a57805739b"} err="failed to get container status \"9a4bdfd391a1f7e7d056226226e430a9e45acc5ded771c27616460a57805739b\": rpc error: code = NotFound desc = could not find container \"9a4bdfd391a1f7e7d056226226e430a9e45acc5ded771c27616460a57805739b\": container with ID starting with 9a4bdfd391a1f7e7d056226226e430a9e45acc5ded771c27616460a57805739b not found: ID does not exist"
Sep 17 16:33:11.751512 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.751452    2574 scope.go:117] "RemoveContainer" containerID="84d8e80a80b000bbf6ce77868fdc02691cc5f647ef06c02754e8ae1a945fd50c"
Sep 17 16:33:11.751718 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:33:11.751692    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d8e80a80b000bbf6ce77868fdc02691cc5f647ef06c02754e8ae1a945fd50c\": container with ID starting with 84d8e80a80b000bbf6ce77868fdc02691cc5f647ef06c02754e8ae1a945fd50c not found: ID does not exist" containerID="84d8e80a80b000bbf6ce77868fdc02691cc5f647ef06c02754e8ae1a945fd50c"
Sep 17 16:33:11.751851 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.751718    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d8e80a80b000bbf6ce77868fdc02691cc5f647ef06c02754e8ae1a945fd50c"} err="failed to get container status \"84d8e80a80b000bbf6ce77868fdc02691cc5f647ef06c02754e8ae1a945fd50c\": rpc error: code = NotFound desc = could not find container \"84d8e80a80b000bbf6ce77868fdc02691cc5f647ef06c02754e8ae1a945fd50c\": container with ID starting with 84d8e80a80b000bbf6ce77868fdc02691cc5f647ef06c02754e8ae1a945fd50c not found: ID does not exist"
Sep 17 16:33:11.759269 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.759234    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"]
Sep 17 16:33:11.765982 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:11.765950    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6c97b497f9-zftql"]
Sep 17 16:33:13.050395 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:13.050349    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bb45650-9553-4f9b-bae1-ec9d4231c294" path="/var/lib/kubelet/pods/2bb45650-9553-4f9b-bae1-ec9d4231c294/volumes"
Sep 17 16:33:15.674861 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:15.674819    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:33:15.797409 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:15.797377    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:33:15.914339 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:15.914297    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:33:16.020266 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:16.020170    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:33:16.171291 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:16.171250    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:33:16.271331 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:16.271239    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:33:17.916164 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:17.916119    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 16:33:18.553344 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:18.553308    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:33:18.628553 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:18.628514    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:33:19.129019 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.128988    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"]
Sep 17 16:33:19.129427 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.129384    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerName="storage-initializer"
Sep 17 16:33:19.129427 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.129396    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerName="storage-initializer"
Sep 17 16:33:19.129427 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.129409    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerName="main"
Sep 17 16:33:19.129427 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.129415    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerName="main"
Sep 17 16:33:19.129596 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.129491    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bb45650-9553-4f9b-bae1-ec9d4231c294" containerName="main"
Sep 17 16:33:19.134431 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.134414    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"
Sep 17 16:33:19.136854 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.136830    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 16:33:19.136978 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.136882    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 16:33:19.137339 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.137319    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-6clcz\""
Sep 17 16:33:19.137463 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.137341    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 16:33:19.140291 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.140272    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"]
Sep 17 16:33:19.236470 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.236412    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:33:19.243514 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.243487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zqvg\" (UniqueName: \"kubernetes.io/projected/dd47f708-2427-499e-b81c-3cf711730589-kube-api-access-4zqvg\") pod \"e2e-pvc-lora-download-hc8dt\" (UID: \"dd47f708-2427-499e-b81c-3cf711730589\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"
Sep 17 16:33:19.344272 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.344230    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqvg\" (UniqueName: \"kubernetes.io/projected/dd47f708-2427-499e-b81c-3cf711730589-kube-api-access-4zqvg\") pod \"e2e-pvc-lora-download-hc8dt\" (UID: \"dd47f708-2427-499e-b81c-3cf711730589\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"
Sep 17 16:33:19.353797 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:19.353768    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zqvg\" (UniqueName: \"kubernetes.io/projected/dd47f708-2427-499e-b81c-3cf711730589-kube-api-access-4zqvg\") pod \"e2e-pvc-lora-download-hc8dt\" (UID: \"dd47f708-2427-499e-b81c-3cf711730589\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"
Sep 17 16:33:21.568515 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:21.568481    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"e2e-pvc-lora-download-hc8dt\" (UID: \"dd47f708-2427-499e-b81c-3cf711730589\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"
Sep 17 16:33:21.575337 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:21.575309    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"e2e-pvc-lora-download-hc8dt\" (UID: \"dd47f708-2427-499e-b81c-3cf711730589\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"
Sep 17 16:33:21.669713 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:21.669682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"e2e-pvc-lora-download-hc8dt\" (UID: \"dd47f708-2427-499e-b81c-3cf711730589\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"
Sep 17 16:33:21.669888 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:21.669819    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"e2e-pvc-lora-download-hc8dt\" (UID: \"dd47f708-2427-499e-b81c-3cf711730589\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"
Sep 17 16:33:21.673498 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:21.673478    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"e2e-pvc-lora-download-hc8dt\" (UID: \"dd47f708-2427-499e-b81c-3cf711730589\") DevicePath \"csi-01ae25341b4ad83b4c37e044ea1363dd9d474f418020ad041b6cdb93016e32be\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"
Sep 17 16:33:21.890342 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:21.890304    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"e2e-pvc-lora-download-hc8dt\" (UID: \"dd47f708-2427-499e-b81c-3cf711730589\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0eac60989084306e19ce440d2a004555fbe7c6909f528bf61ccdf24dbf6bbf4b/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"
Sep 17 16:33:21.922494 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:21.922429    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"e2e-pvc-lora-download-hc8dt\" (UID: \"dd47f708-2427-499e-b81c-3cf711730589\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"
Sep 17 16:33:22.145686 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:22.145601    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"
Sep 17 16:33:22.234396 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:22.234339    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:33:22.329868 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:22.329826    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:33:22.422520 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:22.422486    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"]
Sep 17 16:33:22.425005 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:33:22.424973    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd47f708_2427_499e_b81c_3cf711730589.slice/crio-ef7d6124058686742c990e1cae1d22920ceac39d14c54081e1dac32e29210cd7 WatchSource:0}: Error finding container ef7d6124058686742c990e1cae1d22920ceac39d14c54081e1dac32e29210cd7: Status 404 returned error can't find the container with id ef7d6124058686742c990e1cae1d22920ceac39d14c54081e1dac32e29210cd7
Sep 17 16:33:22.801290 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:22.801209    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt" event={"ID":"dd47f708-2427-499e-b81c-3cf711730589","Type":"ContainerStarted","Data":"d3c5c34fd6fdb47fbf9af70744fbf32ebbf00b3d87c6b880368d1092817c326c"}
Sep 17 16:33:22.801290 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:22.801255    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt" event={"ID":"dd47f708-2427-499e-b81c-3cf711730589","Type":"ContainerStarted","Data":"ef7d6124058686742c990e1cae1d22920ceac39d14c54081e1dac32e29210cd7"}
Sep 17 16:33:22.819361 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:22.819305    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt" podStartSLOduration=6.819288049 podStartE2EDuration="6.819288049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:33:22.817747882 +0000 UTC m=+3268.289587729" watchObservedRunningTime="2026-09-17 16:33:22.819288049 +0000 UTC m=+3268.291127894"
Sep 17 16:33:24.814433 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:24.814398    2574 generic.go:356] "Generic (PLEG): container finished" podID="dd47f708-2427-499e-b81c-3cf711730589" containerID="d3c5c34fd6fdb47fbf9af70744fbf32ebbf00b3d87c6b880368d1092817c326c" exitCode=0
Sep 17 16:33:24.814950 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:24.814467    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt" event={"ID":"dd47f708-2427-499e-b81c-3cf711730589","Type":"ContainerDied","Data":"d3c5c34fd6fdb47fbf9af70744fbf32ebbf00b3d87c6b880368d1092817c326c"}
Sep 17 16:33:25.959294 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:25.959265    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"
Sep 17 16:33:26.114043 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:26.114005    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"dd47f708-2427-499e-b81c-3cf711730589\" (UID: \"dd47f708-2427-499e-b81c-3cf711730589\") "
Sep 17 16:33:26.114244 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:26.114064    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zqvg\" (UniqueName: \"kubernetes.io/projected/dd47f708-2427-499e-b81c-3cf711730589-kube-api-access-4zqvg\") pod \"dd47f708-2427-499e-b81c-3cf711730589\" (UID: \"dd47f708-2427-499e-b81c-3cf711730589\") "
Sep 17 16:33:26.116342 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:26.116311    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd47f708-2427-499e-b81c-3cf711730589-kube-api-access-4zqvg" (OuterVolumeSpecName: "kube-api-access-4zqvg") pod "dd47f708-2427-499e-b81c-3cf711730589" (UID: "dd47f708-2427-499e-b81c-3cf711730589"). InnerVolumeSpecName "kube-api-access-4zqvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:33:26.118727 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:26.118705    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479" (OuterVolumeSpecName: "model-storage") pod "dd47f708-2427-499e-b81c-3cf711730589" (UID: "dd47f708-2427-499e-b81c-3cf711730589"). InnerVolumeSpecName "pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 16:33:26.215365 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:26.215322    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") on node \"ip-10-0-129-37.ec2.internal\" "
Sep 17 16:33:26.215365 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:26.215370    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zqvg\" (UniqueName: \"kubernetes.io/projected/dd47f708-2427-499e-b81c-3cf711730589-kube-api-access-4zqvg\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:33:26.825346 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:26.825313    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"
Sep 17 16:33:26.825566 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:26.825312    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt" event={"ID":"dd47f708-2427-499e-b81c-3cf711730589","Type":"ContainerDied","Data":"ef7d6124058686742c990e1cae1d22920ceac39d14c54081e1dac32e29210cd7"}
Sep 17 16:33:26.825566 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:26.825454    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef7d6124058686742c990e1cae1d22920ceac39d14c54081e1dac32e29210cd7"
Sep 17 16:33:27.188195 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:27.188172    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479") on node "ip-10-0-129-37.ec2.internal"
Sep 17 16:33:27.224496 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:27.224463    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"csi-01ae25341b4ad83b4c37e044ea1363dd9d474f418020ad041b6cdb93016e32be\""
Sep 17 16:33:27.915840 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:27.915789    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 16:33:37.915653 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:37.915611    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 16:33:40.407691 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.407658    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"]
Sep 17 16:33:40.408252 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.408230    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd47f708-2427-499e-b81c-3cf711730589" containerName="storage-initializer"
Sep 17 16:33:40.408340 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.408255    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd47f708-2427-499e-b81c-3cf711730589" containerName="storage-initializer"
Sep 17 16:33:40.408429 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.408387    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd47f708-2427-499e-b81c-3cf711730589" containerName="storage-initializer"
Sep 17 16:33:40.411961 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.411937    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.414959 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.414935    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 16:33:40.415134 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.415108    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:33:40.415494 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.415428    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 16:33:40.415596 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.415536    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-6clcz\""
Sep 17 16:33:40.415664 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.415636    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 17 16:33:40.415664 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.415457    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 16:33:40.429077 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.429052    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.429185 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.429092    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.429185 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.429121    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.429325 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.429206    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-home\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.429325 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.429256    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.429325 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.429287    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-dshm\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.429436 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.429354    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.433329 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.433301    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"]
Sep 17 16:33:40.462685 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.462646    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:33:40.530746 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.530711    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6sk\" (UniqueName: \"kubernetes.io/projected/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-kube-api-access-xc6sk\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.530943 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.530766    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.530943 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.530797    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.530943 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.530922    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.531111 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.531000    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-home\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.531111 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.531047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.531111 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.531076    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-dshm\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.531274 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.531144    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.531274 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.531173    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.531274 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.531224    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.531405 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.531278    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.531405 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.531321    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-home\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.531768 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.531744    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.533507 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.533483    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-dshm\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.533807 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.533790    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.632006 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.631961    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6sk\" (UniqueName: \"kubernetes.io/projected/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-kube-api-access-xc6sk\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:40.641861 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:40.641826    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6sk\" (UniqueName: \"kubernetes.io/projected/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-kube-api-access-xc6sk\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:42.955186 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:42.955137    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:42.961844 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:42.961815    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:43.056641 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:43.056601    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:43.056818 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:43.056733    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:43.063196 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:43.063165    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") DevicePath \"csi-01ae25341b4ad83b4c37e044ea1363dd9d474f418020ad041b6cdb93016e32be\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:43.153776 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:43.153744    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/0eac60989084306e19ce440d2a004555fbe7c6909f528bf61ccdf24dbf6bbf4b/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:43.173869 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:43.173836    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:43.425247 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:43.425201    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:43.712333 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:43.712302    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"]
Sep 17 16:33:43.713419 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:33:43.713390    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdb5c0a_fabf_4b96_803d_35d5f301f9a3.slice/crio-d9ba0d0c3e594093f6a3c737367422cfd49309ab862130bd102e703178b62355 WatchSource:0}: Error finding container d9ba0d0c3e594093f6a3c737367422cfd49309ab862130bd102e703178b62355: Status 404 returned error can't find the container with id d9ba0d0c3e594093f6a3c737367422cfd49309ab862130bd102e703178b62355
Sep 17 16:33:43.915025 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:43.914986    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" event={"ID":"efdb5c0a-fabf-4b96-803d-35d5f301f9a3","Type":"ContainerStarted","Data":"da9ba6cc48f8bf6f6b702d4a11e50b7146cf770ab97cec6707a646b9e241f1c2"}
Sep 17 16:33:43.915223 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:43.915054    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" event={"ID":"efdb5c0a-fabf-4b96-803d-35d5f301f9a3","Type":"ContainerStarted","Data":"d9ba0d0c3e594093f6a3c737367422cfd49309ab862130bd102e703178b62355"}
Sep 17 16:33:46.937866 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:46.937828    2574 generic.go:356] "Generic (PLEG): container finished" podID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerID="da9ba6cc48f8bf6f6b702d4a11e50b7146cf770ab97cec6707a646b9e241f1c2" exitCode=0
Sep 17 16:33:46.938256 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:46.937891    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" event={"ID":"efdb5c0a-fabf-4b96-803d-35d5f301f9a3","Type":"ContainerDied","Data":"da9ba6cc48f8bf6f6b702d4a11e50b7146cf770ab97cec6707a646b9e241f1c2"}
Sep 17 16:33:47.916074 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:47.916018    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 16:33:47.946471 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:47.946413    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" event={"ID":"efdb5c0a-fabf-4b96-803d-35d5f301f9a3","Type":"ContainerStarted","Data":"051ec90d8d302ab24c2bc604fbf6f8006cf20980d17d7e9081507166602267fc"}
Sep 17 16:33:47.973152 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:47.973091    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" podStartSLOduration=5.539158213 podStartE2EDuration="7.97307127s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.433913057s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:33:47.972789659 +0000 UTC m=+3293.444629506" watchObservedRunningTime="2026-09-17 16:33:47.97307127 +0000 UTC m=+3293.444911119"
Sep 17 16:33:48.426128 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:48.426088    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:53.425830 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:53.425783    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:33:53.427863 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:53.427831    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" podUID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 16:33:57.915734 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:33:57.915673    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 16:34:03.426214 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:03.426161    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" podUID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 16:34:07.925957 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:07.925925    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:34:07.933867 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:07.933837    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:34:13.425963 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:13.425909    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" podUID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 16:34:23.426315 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:23.426222    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" podUID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 16:34:27.010482 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:27.010422    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:34:27.644009 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:27.643973    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:34:27.663139 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:27.663105    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"]
Sep 17 16:34:27.675092 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:27.675061    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"]
Sep 17 16:34:27.712608 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:27.712510    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:34:30.391297 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:30.391254    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m" podUID="e329d53d-0466-43f0-a443-654acdb32384" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:34:33.426482 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:33.426410    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" podUID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 16:34:40.390555 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:40.390507    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m" podUID="e329d53d-0466-43f0-a443-654acdb32384" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:34:42.684533 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:42.684422    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m" podUID="e329d53d-0466-43f0-a443-654acdb32384" containerName="main" containerID="cri-o://0d075ef75defbbbd0bb1fcdc4e81954c0668b1145885ae031c3ec4e5e8fa5ba5" gracePeriod=45
Sep 17 16:34:42.700864 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:42.700817    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main" containerID="cri-o://d702811dd14340784a0aa0c34aed4973de5ac89523d02bff54b98e396f307f34" gracePeriod=45
Sep 17 16:34:43.426301 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.426252    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" podUID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 16:34:43.452020 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.451996    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:34:43.630050 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.630022    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-home\") pod \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") "
Sep 17 16:34:43.630258 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.630086    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-model-cache\") pod \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") "
Sep 17 16:34:43.630258 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.630112    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-tls-certs\") pod \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") "
Sep 17 16:34:43.630258 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.630146    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h6mm\" (UniqueName: \"kubernetes.io/projected/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-kube-api-access-6h6mm\") pod \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") "
Sep 17 16:34:43.630258 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.630185    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-tmp-dir\") pod \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") "
Sep 17 16:34:43.630258 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.630218    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-kserve-provision-location\") pod \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") "
Sep 17 16:34:43.630646 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.630273    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-cabundle-cert\") pod \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") "
Sep 17 16:34:43.630646 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.630329    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-dshm\") pod \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\" (UID: \"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20\") "
Sep 17 16:34:43.630646 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.630376    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" (UID: "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:43.630646 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.630635    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:34:43.630855 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.630732    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" (UID: "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:34:43.631028 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.630949    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-home" (OuterVolumeSpecName: "home") pod "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" (UID: "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:43.633254 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.633197    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-kube-api-access-6h6mm" (OuterVolumeSpecName: "kube-api-access-6h6mm") pod "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" (UID: "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20"). InnerVolumeSpecName "kube-api-access-6h6mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:34:43.633373 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.633294    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-dshm" (OuterVolumeSpecName: "dshm") pod "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" (UID: "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:43.633429 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.633379    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" (UID: "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:34:43.649850 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.649811    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" (UID: "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:43.654520 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.654490    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" (UID: "5d0ea49d-2dcf-46e1-9df8-b9154fd71b20"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:43.731430 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.731391    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:34:43.731430 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.731425    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h6mm\" (UniqueName: \"kubernetes.io/projected/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-kube-api-access-6h6mm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:34:43.731834 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.731437    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:34:43.731834 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.731473    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:34:43.731834 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.731487    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:34:43.731834 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.731500    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:34:43.731834 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:43.731512    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:34:44.262780 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:44.262742    2574 generic.go:356] "Generic (PLEG): container finished" podID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerID="d702811dd14340784a0aa0c34aed4973de5ac89523d02bff54b98e396f307f34" exitCode=0
Sep 17 16:34:44.262956 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:44.262798    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"
Sep 17 16:34:44.262956 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:44.262840    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" event={"ID":"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20","Type":"ContainerDied","Data":"d702811dd14340784a0aa0c34aed4973de5ac89523d02bff54b98e396f307f34"}
Sep 17 16:34:44.262956 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:44.262901    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d" event={"ID":"5d0ea49d-2dcf-46e1-9df8-b9154fd71b20","Type":"ContainerDied","Data":"098aa089d792b832631b96d6e967800a003f3502a10b7b38a2f204d492f36143"}
Sep 17 16:34:44.262956 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:44.262935    2574 scope.go:117] "RemoveContainer" containerID="d702811dd14340784a0aa0c34aed4973de5ac89523d02bff54b98e396f307f34"
Sep 17 16:34:44.272777 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:44.272755    2574 scope.go:117] "RemoveContainer" containerID="6e0660003ca33c332db51c9501e45d398f303bad6a623d4e93effe2b2e431eb1"
Sep 17 16:34:44.282159 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:44.282142    2574 scope.go:117] "RemoveContainer" containerID="d702811dd14340784a0aa0c34aed4973de5ac89523d02bff54b98e396f307f34"
Sep 17 16:34:44.282492 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:34:44.282463    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d702811dd14340784a0aa0c34aed4973de5ac89523d02bff54b98e396f307f34\": container with ID starting with d702811dd14340784a0aa0c34aed4973de5ac89523d02bff54b98e396f307f34 not found: ID does not exist" containerID="d702811dd14340784a0aa0c34aed4973de5ac89523d02bff54b98e396f307f34"
Sep 17 16:34:44.282548 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:44.282506    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d702811dd14340784a0aa0c34aed4973de5ac89523d02bff54b98e396f307f34"} err="failed to get container status \"d702811dd14340784a0aa0c34aed4973de5ac89523d02bff54b98e396f307f34\": rpc error: code = NotFound desc = could not find container \"d702811dd14340784a0aa0c34aed4973de5ac89523d02bff54b98e396f307f34\": container with ID starting with d702811dd14340784a0aa0c34aed4973de5ac89523d02bff54b98e396f307f34 not found: ID does not exist"
Sep 17 16:34:44.282548 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:44.282531    2574 scope.go:117] "RemoveContainer" containerID="6e0660003ca33c332db51c9501e45d398f303bad6a623d4e93effe2b2e431eb1"
Sep 17 16:34:44.282793 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:34:44.282775    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e0660003ca33c332db51c9501e45d398f303bad6a623d4e93effe2b2e431eb1\": container with ID starting with 6e0660003ca33c332db51c9501e45d398f303bad6a623d4e93effe2b2e431eb1 not found: ID does not exist" containerID="6e0660003ca33c332db51c9501e45d398f303bad6a623d4e93effe2b2e431eb1"
Sep 17 16:34:44.282835 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:44.282801    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e0660003ca33c332db51c9501e45d398f303bad6a623d4e93effe2b2e431eb1"} err="failed to get container status \"6e0660003ca33c332db51c9501e45d398f303bad6a623d4e93effe2b2e431eb1\": rpc error: code = NotFound desc = could not find container \"6e0660003ca33c332db51c9501e45d398f303bad6a623d4e93effe2b2e431eb1\": container with ID starting with 6e0660003ca33c332db51c9501e45d398f303bad6a623d4e93effe2b2e431eb1 not found: ID does not exist"
Sep 17 16:34:44.300265 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:44.300242    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"]
Sep 17 16:34:44.304342 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:44.304320    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-76bd6ff677-88h4d"]
Sep 17 16:34:45.049271 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:45.049237    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" path="/var/lib/kubelet/pods/5d0ea49d-2dcf-46e1-9df8-b9154fd71b20/volumes"
Sep 17 16:34:50.390848 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:50.390805    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m" podUID="e329d53d-0466-43f0-a443-654acdb32384" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:34:50.391218 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:50.390922    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"
Sep 17 16:34:53.426762 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:53.426711    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" podUID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 16:34:53.631245 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:53.631206    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:34:53.708219 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:34:53.708123    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:00.390635 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:00.390590    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m" podUID="e329d53d-0466-43f0-a443-654acdb32384" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:35:03.435745 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:03.435715    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:35:03.443528 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:03.443499    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:35:10.390931 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:10.390882    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m" podUID="e329d53d-0466-43f0-a443-654acdb32384" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:35:12.931650 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:12.931621    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"
Sep 17 16:35:12.999174 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:12.999088    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9xhw\" (UniqueName: \"kubernetes.io/projected/e329d53d-0466-43f0-a443-654acdb32384-kube-api-access-p9xhw\") pod \"e329d53d-0466-43f0-a443-654acdb32384\" (UID: \"e329d53d-0466-43f0-a443-654acdb32384\") "
Sep 17 16:35:12.999174 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:12.999125    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e329d53d-0466-43f0-a443-654acdb32384-tls-certs\") pod \"e329d53d-0466-43f0-a443-654acdb32384\" (UID: \"e329d53d-0466-43f0-a443-654acdb32384\") "
Sep 17 16:35:13.001299 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:13.001266    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e329d53d-0466-43f0-a443-654acdb32384-kube-api-access-p9xhw" (OuterVolumeSpecName: "kube-api-access-p9xhw") pod "e329d53d-0466-43f0-a443-654acdb32384" (UID: "e329d53d-0466-43f0-a443-654acdb32384"). InnerVolumeSpecName "kube-api-access-p9xhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:35:13.001434 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:13.001313    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e329d53d-0466-43f0-a443-654acdb32384-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e329d53d-0466-43f0-a443-654acdb32384" (UID: "e329d53d-0466-43f0-a443-654acdb32384"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:35:13.100540 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:13.100509    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9xhw\" (UniqueName: \"kubernetes.io/projected/e329d53d-0466-43f0-a443-654acdb32384-kube-api-access-p9xhw\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:35:13.100540 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:13.100536    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e329d53d-0466-43f0-a443-654acdb32384-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:35:13.420117 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:13.420084    2574 generic.go:356] "Generic (PLEG): container finished" podID="e329d53d-0466-43f0-a443-654acdb32384" containerID="0d075ef75defbbbd0bb1fcdc4e81954c0668b1145885ae031c3ec4e5e8fa5ba5" exitCode=0
Sep 17 16:35:13.420310 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:13.420125    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"
Sep 17 16:35:13.420310 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:13.420173    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m" event={"ID":"e329d53d-0466-43f0-a443-654acdb32384","Type":"ContainerDied","Data":"0d075ef75defbbbd0bb1fcdc4e81954c0668b1145885ae031c3ec4e5e8fa5ba5"}
Sep 17 16:35:13.420310 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:13.420223    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m" event={"ID":"e329d53d-0466-43f0-a443-654acdb32384","Type":"ContainerDied","Data":"1f9e824816010c4036509000a648d57518cfafc57470b4174d1c9e041e5ec11f"}
Sep 17 16:35:13.420310 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:13.420245    2574 scope.go:117] "RemoveContainer" containerID="0d075ef75defbbbd0bb1fcdc4e81954c0668b1145885ae031c3ec4e5e8fa5ba5"
Sep 17 16:35:13.429714 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:13.429666    2574 scope.go:117] "RemoveContainer" containerID="0d075ef75defbbbd0bb1fcdc4e81954c0668b1145885ae031c3ec4e5e8fa5ba5"
Sep 17 16:35:13.429994 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:35:13.429968    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d075ef75defbbbd0bb1fcdc4e81954c0668b1145885ae031c3ec4e5e8fa5ba5\": container with ID starting with 0d075ef75defbbbd0bb1fcdc4e81954c0668b1145885ae031c3ec4e5e8fa5ba5 not found: ID does not exist" containerID="0d075ef75defbbbd0bb1fcdc4e81954c0668b1145885ae031c3ec4e5e8fa5ba5"
Sep 17 16:35:13.430097 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:13.430000    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d075ef75defbbbd0bb1fcdc4e81954c0668b1145885ae031c3ec4e5e8fa5ba5"} err="failed to get container status \"0d075ef75defbbbd0bb1fcdc4e81954c0668b1145885ae031c3ec4e5e8fa5ba5\": rpc error: code = NotFound desc = could not find container \"0d075ef75defbbbd0bb1fcdc4e81954c0668b1145885ae031c3ec4e5e8fa5ba5\": container with ID starting with 0d075ef75defbbbd0bb1fcdc4e81954c0668b1145885ae031c3ec4e5e8fa5ba5 not found: ID does not exist"
Sep 17 16:35:13.444891 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:13.444863    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"]
Sep 17 16:35:13.454832 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:13.454811    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b459p22m"]
Sep 17 16:35:15.049039 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:15.049008    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e329d53d-0466-43f0-a443-654acdb32384" path="/var/lib/kubelet/pods/e329d53d-0466-43f0-a443-654acdb32384/volumes"
Sep 17 16:35:17.795241 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:17.795196    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:17.885689 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:17.885651    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:22.863259 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:22.863220    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:29.119240 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:29.119199    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx_efdb5c0a-fabf-4b96-803d-35d5f301f9a3/storage-initializer/0.log"
Sep 17 16:35:29.130839 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:29.130812    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx_efdb5c0a-fabf-4b96-803d-35d5f301f9a3/main/0.log"
Sep 17 16:35:33.959724 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:33.959679    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:38.854955 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:38.854921    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:40.058200 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:40.058161    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:46.156390 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:46.156353    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:48.709326 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:48.709241    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"]
Sep 17 16:35:48.747780 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:48.747747    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:55.281585 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:35:55.281528    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:01.745641 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:01.745600    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:01.813563 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:01.813526    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:03.732234 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:03.732191    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" podUID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerName="main" containerID="cri-o://051ec90d8d302ab24c2bc604fbf6f8006cf20980d17d7e9081507166602267fc" gracePeriod=45
Sep 17 16:36:04.389795 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.389767    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:36:04.450756 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.450674    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc6sk\" (UniqueName: \"kubernetes.io/projected/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-kube-api-access-xc6sk\") pod \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") "
Sep 17 16:36:04.450890 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.450825    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") pod \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") "
Sep 17 16:36:04.450890 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.450868    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-tmp-dir\") pod \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") "
Sep 17 16:36:04.451010 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.450893    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-home\") pod \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") "
Sep 17 16:36:04.451010 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.450960    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-cabundle-cert\") pod \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") "
Sep 17 16:36:04.451010 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.451004    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-tls-certs\") pod \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") "
Sep 17 16:36:04.451165 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.451043    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-dshm\") pod \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") "
Sep 17 16:36:04.451165 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.451096    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-model-cache\") pod \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") "
Sep 17 16:36:04.451165 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.451131    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-kserve-provision-location\") pod \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\" (UID: \"efdb5c0a-fabf-4b96-803d-35d5f301f9a3\") "
Sep 17 16:36:04.451587 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.451476    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "efdb5c0a-fabf-4b96-803d-35d5f301f9a3" (UID: "efdb5c0a-fabf-4b96-803d-35d5f301f9a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:36:04.451738 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.451666    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "efdb5c0a-fabf-4b96-803d-35d5f301f9a3" (UID: "efdb5c0a-fabf-4b96-803d-35d5f301f9a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:36:04.451911 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.451893    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-home" (OuterVolumeSpecName: "home") pod "efdb5c0a-fabf-4b96-803d-35d5f301f9a3" (UID: "efdb5c0a-fabf-4b96-803d-35d5f301f9a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:36:04.453484 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.453428    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-kube-api-access-xc6sk" (OuterVolumeSpecName: "kube-api-access-xc6sk") pod "efdb5c0a-fabf-4b96-803d-35d5f301f9a3" (UID: "efdb5c0a-fabf-4b96-803d-35d5f301f9a3"). InnerVolumeSpecName "kube-api-access-xc6sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:36:04.453665 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.453627    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-dshm" (OuterVolumeSpecName: "dshm") pod "efdb5c0a-fabf-4b96-803d-35d5f301f9a3" (UID: "efdb5c0a-fabf-4b96-803d-35d5f301f9a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:36:04.453854 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.453836    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efdb5c0a-fabf-4b96-803d-35d5f301f9a3" (UID: "efdb5c0a-fabf-4b96-803d-35d5f301f9a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:36:04.456058 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.456037    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "efdb5c0a-fabf-4b96-803d-35d5f301f9a3" (UID: "efdb5c0a-fabf-4b96-803d-35d5f301f9a3"). InnerVolumeSpecName "pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 16:36:04.464717 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.464691    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efdb5c0a-fabf-4b96-803d-35d5f301f9a3" (UID: "efdb5c0a-fabf-4b96-803d-35d5f301f9a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:36:04.475759 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.475738    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "efdb5c0a-fabf-4b96-803d-35d5f301f9a3" (UID: "efdb5c0a-fabf-4b96-803d-35d5f301f9a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:36:04.551935 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.551896    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc6sk\" (UniqueName: \"kubernetes.io/projected/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-kube-api-access-xc6sk\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:36:04.552088 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.551947    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") on node \"ip-10-0-129-37.ec2.internal\" "
Sep 17 16:36:04.552088 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.551957    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:36:04.552088 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.551967    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:36:04.552088 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.551977    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:36:04.552088 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.551986    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:36:04.552088 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.551993    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:36:04.552088 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.552001    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:36:04.552088 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.552009    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efdb5c0a-fabf-4b96-803d-35d5f301f9a3-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:36:04.562094 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.562075    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479") on node "ip-10-0-129-37.ec2.internal"
Sep 17 16:36:04.653227 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.653177    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-ebcb778c-62f8-4dca-a0d2-db71e2a7b7a5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-07583f79c022a7479\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"csi-01ae25341b4ad83b4c37e044ea1363dd9d474f418020ad041b6cdb93016e32be\""
Sep 17 16:36:04.677311 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.677276    2574 generic.go:356] "Generic (PLEG): container finished" podID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerID="051ec90d8d302ab24c2bc604fbf6f8006cf20980d17d7e9081507166602267fc" exitCode=0
Sep 17 16:36:04.677490 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.677330    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"
Sep 17 16:36:04.677490 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.677364    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" event={"ID":"efdb5c0a-fabf-4b96-803d-35d5f301f9a3","Type":"ContainerDied","Data":"051ec90d8d302ab24c2bc604fbf6f8006cf20980d17d7e9081507166602267fc"}
Sep 17 16:36:04.677490 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.677485    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx" event={"ID":"efdb5c0a-fabf-4b96-803d-35d5f301f9a3","Type":"ContainerDied","Data":"d9ba0d0c3e594093f6a3c737367422cfd49309ab862130bd102e703178b62355"}
Sep 17 16:36:04.677629 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.677514    2574 scope.go:117] "RemoveContainer" containerID="051ec90d8d302ab24c2bc604fbf6f8006cf20980d17d7e9081507166602267fc"
Sep 17 16:36:04.687364 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.687344    2574 scope.go:117] "RemoveContainer" containerID="da9ba6cc48f8bf6f6b702d4a11e50b7146cf770ab97cec6707a646b9e241f1c2"
Sep 17 16:36:04.695532 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.695508    2574 scope.go:117] "RemoveContainer" containerID="051ec90d8d302ab24c2bc604fbf6f8006cf20980d17d7e9081507166602267fc"
Sep 17 16:36:04.695822 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:36:04.695798    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051ec90d8d302ab24c2bc604fbf6f8006cf20980d17d7e9081507166602267fc\": container with ID starting with 051ec90d8d302ab24c2bc604fbf6f8006cf20980d17d7e9081507166602267fc not found: ID does not exist" containerID="051ec90d8d302ab24c2bc604fbf6f8006cf20980d17d7e9081507166602267fc"
Sep 17 16:36:04.695875 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.695831    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051ec90d8d302ab24c2bc604fbf6f8006cf20980d17d7e9081507166602267fc"} err="failed to get container status \"051ec90d8d302ab24c2bc604fbf6f8006cf20980d17d7e9081507166602267fc\": rpc error: code = NotFound desc = could not find container \"051ec90d8d302ab24c2bc604fbf6f8006cf20980d17d7e9081507166602267fc\": container with ID starting with 051ec90d8d302ab24c2bc604fbf6f8006cf20980d17d7e9081507166602267fc not found: ID does not exist"
Sep 17 16:36:04.695875 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.695851    2574 scope.go:117] "RemoveContainer" containerID="da9ba6cc48f8bf6f6b702d4a11e50b7146cf770ab97cec6707a646b9e241f1c2"
Sep 17 16:36:04.696095 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:36:04.696075    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da9ba6cc48f8bf6f6b702d4a11e50b7146cf770ab97cec6707a646b9e241f1c2\": container with ID starting with da9ba6cc48f8bf6f6b702d4a11e50b7146cf770ab97cec6707a646b9e241f1c2 not found: ID does not exist" containerID="da9ba6cc48f8bf6f6b702d4a11e50b7146cf770ab97cec6707a646b9e241f1c2"
Sep 17 16:36:04.696155 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.696103    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da9ba6cc48f8bf6f6b702d4a11e50b7146cf770ab97cec6707a646b9e241f1c2"} err="failed to get container status \"da9ba6cc48f8bf6f6b702d4a11e50b7146cf770ab97cec6707a646b9e241f1c2\": rpc error: code = NotFound desc = could not find container \"da9ba6cc48f8bf6f6b702d4a11e50b7146cf770ab97cec6707a646b9e241f1c2\": container with ID starting with da9ba6cc48f8bf6f6b702d4a11e50b7146cf770ab97cec6707a646b9e241f1c2 not found: ID does not exist"
Sep 17 16:36:04.711518 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.711428    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"]
Sep 17 16:36:04.719710 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:04.719676    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5b755b667d-zkgcx"]
Sep 17 16:36:05.049919 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:05.049839    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" path="/var/lib/kubelet/pods/efdb5c0a-fabf-4b96-803d-35d5f301f9a3/volumes"
Sep 17 16:36:05.709008 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:05.708974    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:05.829251 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:05.829215    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:05.939165 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:05.939118    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:06.477609 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:06.477570    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:06.586128 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:06.586096    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:08.380609 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:08.380560    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:08.482788 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:08.482754    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:09.094818 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.094784    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"]
Sep 17 16:36:09.095158 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.095146    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e329d53d-0466-43f0-a443-654acdb32384" containerName="main"
Sep 17 16:36:09.095208 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.095160    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e329d53d-0466-43f0-a443-654acdb32384" containerName="main"
Sep 17 16:36:09.095208 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.095170    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main"
Sep 17 16:36:09.095208 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.095176    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main"
Sep 17 16:36:09.095208 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.095183    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerName="storage-initializer"
Sep 17 16:36:09.095208 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.095188    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerName="storage-initializer"
Sep 17 16:36:09.095208 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.095199    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerName="main"
Sep 17 16:36:09.095208 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.095204    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerName="main"
Sep 17 16:36:09.095208 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.095211    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="storage-initializer"
Sep 17 16:36:09.095470 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.095216    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="storage-initializer"
Sep 17 16:36:09.095470 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.095279    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d0ea49d-2dcf-46e1-9df8-b9154fd71b20" containerName="main"
Sep 17 16:36:09.095470 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.095288    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e329d53d-0466-43f0-a443-654acdb32384" containerName="main"
Sep 17 16:36:09.095470 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.095294    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="efdb5c0a-fabf-4b96-803d-35d5f301f9a3" containerName="main"
Sep 17 16:36:09.097163 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.097147    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"
Sep 17 16:36:09.099728 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.099702    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-z4cq6\""
Sep 17 16:36:09.099863 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.099709    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 16:36:09.100222 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.100206    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 17 16:36:09.100274 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.100208    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 16:36:09.107576 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.107553    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"]
Sep 17 16:36:09.193710 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.193678    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7cb\" (UniqueName: \"kubernetes.io/projected/ca3eb25d-38dc-4768-90e9-5b86bc60133d-kube-api-access-xf7cb\") pod \"e2e-pvc-lora-download-whghw\" (UID: \"ca3eb25d-38dc-4768-90e9-5b86bc60133d\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"
Sep 17 16:36:09.227997 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.227964    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:09.294527 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.294490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7cb\" (UniqueName: \"kubernetes.io/projected/ca3eb25d-38dc-4768-90e9-5b86bc60133d-kube-api-access-xf7cb\") pod \"e2e-pvc-lora-download-whghw\" (UID: \"ca3eb25d-38dc-4768-90e9-5b86bc60133d\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"
Sep 17 16:36:09.303490 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.303461    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7cb\" (UniqueName: \"kubernetes.io/projected/ca3eb25d-38dc-4768-90e9-5b86bc60133d-kube-api-access-xf7cb\") pod \"e2e-pvc-lora-download-whghw\" (UID: \"ca3eb25d-38dc-4768-90e9-5b86bc60133d\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"
Sep 17 16:36:09.896124 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.896086    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:09.996264 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:09.996226    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:10.689252 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:10.689219    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"]
Sep 17 16:36:10.692291 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:10.692263    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-hc8dt"]
Sep 17 16:36:11.049667 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:11.049587    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd47f708-2427-499e-b81c-3cf711730589" path="/var/lib/kubelet/pods/dd47f708-2427-499e-b81c-3cf711730589/volumes"
Sep 17 16:36:12.031142 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:12.031104    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:12.131341 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:12.131301    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:12.944641 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:12.944603    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:13.933744 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:13.933718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"e2e-pvc-lora-download-whghw\" (UID: \"ca3eb25d-38dc-4768-90e9-5b86bc60133d\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"
Sep 17 16:36:13.940510 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:13.940486    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"e2e-pvc-lora-download-whghw\" (UID: \"ca3eb25d-38dc-4768-90e9-5b86bc60133d\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"
Sep 17 16:36:14.034692 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:14.034657    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"e2e-pvc-lora-download-whghw\" (UID: \"ca3eb25d-38dc-4768-90e9-5b86bc60133d\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"
Sep 17 16:36:14.034869 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:14.034796    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"e2e-pvc-lora-download-whghw\" (UID: \"ca3eb25d-38dc-4768-90e9-5b86bc60133d\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"
Sep 17 16:36:14.037865 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:14.037835    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"e2e-pvc-lora-download-whghw\" (UID: \"ca3eb25d-38dc-4768-90e9-5b86bc60133d\") DevicePath \"csi-6891c468deb24d70da76a392ac947aee09200c4f0a2c290fab5668a762e57dfe\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"
Sep 17 16:36:14.247680 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:14.247650    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"e2e-pvc-lora-download-whghw\" (UID: \"ca3eb25d-38dc-4768-90e9-5b86bc60133d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/68824b57b9063840d240d8ffa72a9854505f72b49696d175d38d2d60f44ef4a8/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"
Sep 17 16:36:14.267842 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:14.267804    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"e2e-pvc-lora-download-whghw\" (UID: \"ca3eb25d-38dc-4768-90e9-5b86bc60133d\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"
Sep 17 16:36:14.506418 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:14.506339    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"
Sep 17 16:36:14.782763 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:14.782725    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"]
Sep 17 16:36:14.783940 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:36:14.783910    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca3eb25d_38dc_4768_90e9_5b86bc60133d.slice/crio-9d1b5132dc5d99fb107008421c32ba5b0f6c1d00655d71b53b3eebef957da502 WatchSource:0}: Error finding container 9d1b5132dc5d99fb107008421c32ba5b0f6c1d00655d71b53b3eebef957da502: Status 404 returned error can't find the container with id 9d1b5132dc5d99fb107008421c32ba5b0f6c1d00655d71b53b3eebef957da502
Sep 17 16:36:15.742489 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:15.742426    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw" event={"ID":"ca3eb25d-38dc-4768-90e9-5b86bc60133d","Type":"ContainerStarted","Data":"e3530083d3fb637e3011f7b710821bd3b0ac8115ce1364b57fffa5dbe2835014"}
Sep 17 16:36:15.742987 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:15.742504    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw" event={"ID":"ca3eb25d-38dc-4768-90e9-5b86bc60133d","Type":"ContainerStarted","Data":"9d1b5132dc5d99fb107008421c32ba5b0f6c1d00655d71b53b3eebef957da502"}
Sep 17 16:36:15.778521 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:15.778424    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw" podStartSLOduration=9.778398082 podStartE2EDuration="9.778398082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:36:15.775418265 +0000 UTC m=+3441.247258129" watchObservedRunningTime="2026-09-17 16:36:15.778398082 +0000 UTC m=+3441.250237929"
Sep 17 16:36:17.753517 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:17.753483    2574 generic.go:356] "Generic (PLEG): container finished" podID="ca3eb25d-38dc-4768-90e9-5b86bc60133d" containerID="e3530083d3fb637e3011f7b710821bd3b0ac8115ce1364b57fffa5dbe2835014" exitCode=0
Sep 17 16:36:17.753946 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:17.753559    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw" event={"ID":"ca3eb25d-38dc-4768-90e9-5b86bc60133d","Type":"ContainerDied","Data":"e3530083d3fb637e3011f7b710821bd3b0ac8115ce1364b57fffa5dbe2835014"}
Sep 17 16:36:18.435228 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:18.435195    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:18.542004 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:18.541965    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:18.896621 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:18.896598    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"
Sep 17 16:36:18.978680 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:18.978649    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"ca3eb25d-38dc-4768-90e9-5b86bc60133d\" (UID: \"ca3eb25d-38dc-4768-90e9-5b86bc60133d\") "
Sep 17 16:36:18.978848 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:18.978688    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf7cb\" (UniqueName: \"kubernetes.io/projected/ca3eb25d-38dc-4768-90e9-5b86bc60133d-kube-api-access-xf7cb\") pod \"ca3eb25d-38dc-4768-90e9-5b86bc60133d\" (UID: \"ca3eb25d-38dc-4768-90e9-5b86bc60133d\") "
Sep 17 16:36:18.980998 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:18.980967    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca3eb25d-38dc-4768-90e9-5b86bc60133d-kube-api-access-xf7cb" (OuterVolumeSpecName: "kube-api-access-xf7cb") pod "ca3eb25d-38dc-4768-90e9-5b86bc60133d" (UID: "ca3eb25d-38dc-4768-90e9-5b86bc60133d"). InnerVolumeSpecName "kube-api-access-xf7cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:36:18.981588 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:18.981566    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706" (OuterVolumeSpecName: "model-storage") pod "ca3eb25d-38dc-4768-90e9-5b86bc60133d" (UID: "ca3eb25d-38dc-4768-90e9-5b86bc60133d"). InnerVolumeSpecName "pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 16:36:19.080132 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:19.080102    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") on node \"ip-10-0-129-37.ec2.internal\" "
Sep 17 16:36:19.080132 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:19.080130    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf7cb\" (UniqueName: \"kubernetes.io/projected/ca3eb25d-38dc-4768-90e9-5b86bc60133d-kube-api-access-xf7cb\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:36:19.220040 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:19.220005    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:19.321841 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:19.321750    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:19.765039 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:19.765005    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw" event={"ID":"ca3eb25d-38dc-4768-90e9-5b86bc60133d","Type":"ContainerDied","Data":"9d1b5132dc5d99fb107008421c32ba5b0f6c1d00655d71b53b3eebef957da502"}
Sep 17 16:36:19.765039 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:19.765044    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d1b5132dc5d99fb107008421c32ba5b0f6c1d00655d71b53b3eebef957da502"
Sep 17 16:36:19.765243 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:19.765021    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"
Sep 17 16:36:20.049424 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:20.049397    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706") on node "ip-10-0-129-37.ec2.internal"
Sep 17 16:36:20.088346 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:20.088310    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"csi-6891c468deb24d70da76a392ac947aee09200c4f0a2c290fab5668a762e57dfe\""
Sep 17 16:36:48.294508 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.294473    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"]
Sep 17 16:36:48.295090 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.295046    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca3eb25d-38dc-4768-90e9-5b86bc60133d" containerName="storage-initializer"
Sep 17 16:36:48.295090 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.295068    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca3eb25d-38dc-4768-90e9-5b86bc60133d" containerName="storage-initializer"
Sep 17 16:36:48.295215 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.295180    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca3eb25d-38dc-4768-90e9-5b86bc60133d" containerName="storage-initializer"
Sep 17 16:36:48.298376 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.298352    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.300735 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.300716    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-z4cq6\""
Sep 17 16:36:48.300969 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.300954    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 16:36:48.301035 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.300996    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 16:36:48.301430 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.301413    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 17 16:36:48.318261 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.318237    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"]
Sep 17 16:36:48.320757 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.320724    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:36:48.431541 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.431500    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.431541 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.431544    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.431783 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.431610    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.431783 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.431633    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.431783 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.431687    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.431783 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.431717    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chvlf\" (UniqueName: \"kubernetes.io/projected/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-kube-api-access-chvlf\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.533051 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.533007    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.533051 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.533057    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.533309 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.533099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.533309 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.533127    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chvlf\" (UniqueName: \"kubernetes.io/projected/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-kube-api-access-chvlf\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.533309 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.533200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.533309 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.533236    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.533615 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.533594    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.533685 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.533613    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.533685 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.533665    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.535702 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.535669    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.535921 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.535904    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:48.544808 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:48.544752    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chvlf\" (UniqueName: \"kubernetes.io/projected/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-kube-api-access-chvlf\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:55.297361 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:55.297322    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:55.304812 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:55.304783    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:55.398600 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:55.398569    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:55.398770 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:55.398707    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:55.401705 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:55.401679    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") DevicePath \"csi-6891c468deb24d70da76a392ac947aee09200c4f0a2c290fab5668a762e57dfe\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:55.490054 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:55.490024    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/68824b57b9063840d240d8ffa72a9854505f72b49696d175d38d2d60f44ef4a8/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:55.510286 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:55.510260    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:55.811934 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:55.811904    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-z4cq6\""
Sep 17 16:36:55.819725 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:55.819705    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:56.099626 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:56.099507    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"]
Sep 17 16:36:56.102124 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:36:56.102093    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509a3b6f_24b8_4bfd_a6eb_e5c973c4ef30.slice/crio-64700f0fccc75a09680dd486c1bd919312ffd9dada7a27ec80e614fce1d231fe WatchSource:0}: Error finding container 64700f0fccc75a09680dd486c1bd919312ffd9dada7a27ec80e614fce1d231fe: Status 404 returned error can't find the container with id 64700f0fccc75a09680dd486c1bd919312ffd9dada7a27ec80e614fce1d231fe
Sep 17 16:36:56.950067 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:56.950022    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5" event={"ID":"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30","Type":"ContainerStarted","Data":"4ac8943e63125063ee35eb81bf8deed4361e121961e3819b4c3f8ced7539a7b9"}
Sep 17 16:36:56.950536 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:56.950095    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5" event={"ID":"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30","Type":"ContainerStarted","Data":"64700f0fccc75a09680dd486c1bd919312ffd9dada7a27ec80e614fce1d231fe"}
Sep 17 16:36:56.974560 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:56.974506    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5" podStartSLOduration=8.974489363 podStartE2EDuration="8.974489363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:36:56.972376335 +0000 UTC m=+3482.444216196" watchObservedRunningTime="2026-09-17 16:36:56.974489363 +0000 UTC m=+3482.446329225"
Sep 17 16:36:57.820487 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:57.820437    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:36:59.831362 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:36:59.831253    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:37:05.820548 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:05.820497    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:37:05.822262 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:05.822219    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5" podUID="509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 16:37:15.820893 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:15.820840    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5" podUID="509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 16:37:25.820936 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:25.820843    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5" podUID="509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 16:37:28.984740 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:28.984704    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:37:35.821217 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:35.821171    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5" podUID="509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 16:37:45.820412 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:45.820360    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5" podUID="509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 16:37:48.952819 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:48.952784    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:37:49.052627 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:49.052586    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:37:55.820371 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:55.820323    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5" podUID="509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 16:37:56.214705 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:56.214668    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:37:57.420546 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.420506    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:37:57.926563 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.926518    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"]
Sep 17 16:37:57.938008 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.937971    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 16:37:57.938245 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.938083    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"
Sep 17 16:37:57.941910 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.941880    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 16:37:57.942067 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.942001    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-hjkwx\""
Sep 17 16:37:57.942170 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.942134    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:57.942547 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.942526    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 16:37:57.944221 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.944199    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 16:37:57.945604 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.945580    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:37:57.946163 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.946142    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-dcl7n\""
Sep 17 16:37:57.946163 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.946154    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-cqc55\""
Sep 17 16:37:57.948837 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.948812    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"]
Sep 17 16:37:57.952620 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.952587    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 16:37:57.964283 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:57.964251    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 16:37:58.000258 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.000213    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.000460 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.000275    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.000460 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.000333    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.000460 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.000395    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a92990-f687-4d47-b8a4-9a09d76d2d94-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.000610 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.000460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.000610 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.000484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5884\" (UniqueName: \"kubernetes.io/projected/43e7dd49-495c-4ea0-9a26-fab1c604f712-kube-api-access-s5884\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4\" (UID: \"43e7dd49-495c-4ea0-9a26-fab1c604f712\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"
Sep 17 16:37:58.000610 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.000507    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.000610 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.000545    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a92990-f687-4d47-b8a4-9a09d76d2d94-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.000610 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.000572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjw4\" (UniqueName: \"kubernetes.io/projected/90a92990-f687-4d47-b8a4-9a09d76d2d94-kube-api-access-dxjw4\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.000610 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.000587    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e7dd49-495c-4ea0-9a26-fab1c604f712-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4\" (UID: \"43e7dd49-495c-4ea0-9a26-fab1c604f712\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"
Sep 17 16:37:58.101764 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.101727    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.101764 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.101772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.102035 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.101801    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.102035 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.101934    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a92990-f687-4d47-b8a4-9a09d76d2d94-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.102035 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.102023    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.102198 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.102053    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5884\" (UniqueName: \"kubernetes.io/projected/43e7dd49-495c-4ea0-9a26-fab1c604f712-kube-api-access-s5884\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4\" (UID: \"43e7dd49-495c-4ea0-9a26-fab1c604f712\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"
Sep 17 16:37:58.102198 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.102089    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.102198 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.102136    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a92990-f687-4d47-b8a4-9a09d76d2d94-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.102198 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.102179    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjw4\" (UniqueName: \"kubernetes.io/projected/90a92990-f687-4d47-b8a4-9a09d76d2d94-kube-api-access-dxjw4\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.102407 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.102205    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e7dd49-495c-4ea0-9a26-fab1c604f712-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4\" (UID: \"43e7dd49-495c-4ea0-9a26-fab1c604f712\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"
Sep 17 16:37:58.102407 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.102223    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.102407 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.102262    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.102407 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.102317    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.102652 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.102580    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a92990-f687-4d47-b8a4-9a09d76d2d94-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.102652 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.102604    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.104966 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.104920    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a92990-f687-4d47-b8a4-9a09d76d2d94-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.105141 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.105049    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.105322 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.105290    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e7dd49-495c-4ea0-9a26-fab1c604f712-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4\" (UID: \"43e7dd49-495c-4ea0-9a26-fab1c604f712\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"
Sep 17 16:37:58.111839 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.111812    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjw4\" (UniqueName: \"kubernetes.io/projected/90a92990-f687-4d47-b8a4-9a09d76d2d94-kube-api-access-dxjw4\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.112461 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.112418    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5884\" (UniqueName: \"kubernetes.io/projected/43e7dd49-495c-4ea0-9a26-fab1c604f712-kube-api-access-s5884\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4\" (UID: \"43e7dd49-495c-4ea0-9a26-fab1c604f712\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"
Sep 17 16:37:58.252699 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.252594    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"
Sep 17 16:37:58.260787 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.260757    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:37:58.555900 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.555872    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"]
Sep 17 16:37:58.558758 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:37:58.558727    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e7dd49_495c_4ea0_9a26_fab1c604f712.slice/crio-8bdc7ce7d3c12286f0dd1a859cb32cdedaa91690e179704c37b9aea51ef56a5a WatchSource:0}: Error finding container 8bdc7ce7d3c12286f0dd1a859cb32cdedaa91690e179704c37b9aea51ef56a5a: Status 404 returned error can't find the container with id 8bdc7ce7d3c12286f0dd1a859cb32cdedaa91690e179704c37b9aea51ef56a5a
Sep 17 16:37:58.563117 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:58.563082    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 16:37:58.565910 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:37:58.565884    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a92990_f687_4d47_b8a4_9a09d76d2d94.slice/crio-6fe1d2b7c8662413273936a33159ae41014b2e388071694163297ffa747d239e WatchSource:0}: Error finding container 6fe1d2b7c8662413273936a33159ae41014b2e388071694163297ffa747d239e: Status 404 returned error can't find the container with id 6fe1d2b7c8662413273936a33159ae41014b2e388071694163297ffa747d239e
Sep 17 16:37:59.269606 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:59.269503    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"90a92990-f687-4d47-b8a4-9a09d76d2d94","Type":"ContainerStarted","Data":"cc054125eb7e730cb9b7734019436d2c915ff1e89bd9aa951f2d5fd16d95b70d"}
Sep 17 16:37:59.269818 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:59.269659    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"90a92990-f687-4d47-b8a4-9a09d76d2d94","Type":"ContainerStarted","Data":"6fe1d2b7c8662413273936a33159ae41014b2e388071694163297ffa747d239e"}
Sep 17 16:37:59.271501 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:59.271472    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4" event={"ID":"43e7dd49-495c-4ea0-9a26-fab1c604f712","Type":"ContainerStarted","Data":"0b8561d6f6f55ac71e97d9b77fb9afa5aa25f4ba460833536beff0037f4566ec"}
Sep 17 16:37:59.271661 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:59.271516    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4" event={"ID":"43e7dd49-495c-4ea0-9a26-fab1c604f712","Type":"ContainerStarted","Data":"8bdc7ce7d3c12286f0dd1a859cb32cdedaa91690e179704c37b9aea51ef56a5a"}
Sep 17 16:37:59.271661 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:59.271568    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"
Sep 17 16:37:59.315329 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:37:59.315273    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4" podStartSLOduration=2.315255575 podStartE2EDuration="2.315255575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:37:59.312770442 +0000 UTC m=+3544.784610288" watchObservedRunningTime="2026-09-17 16:37:59.315255575 +0000 UTC m=+3544.787095421"
Sep 17 16:38:00.351918 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:00.351878    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:38:01.285987 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:01.285953    2574 generic.go:356] "Generic (PLEG): container finished" podID="90a92990-f687-4d47-b8a4-9a09d76d2d94" containerID="cc054125eb7e730cb9b7734019436d2c915ff1e89bd9aa951f2d5fd16d95b70d" exitCode=0
Sep 17 16:38:01.286181 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:01.286033    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"90a92990-f687-4d47-b8a4-9a09d76d2d94","Type":"ContainerDied","Data":"cc054125eb7e730cb9b7734019436d2c915ff1e89bd9aa951f2d5fd16d95b70d"}
Sep 17 16:38:02.294110 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:02.294074    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"90a92990-f687-4d47-b8a4-9a09d76d2d94","Type":"ContainerStarted","Data":"3e3aa068db87d756459b2ec265920b0d0f36d43dc09ae786bf179a4950f75006"}
Sep 17 16:38:02.323676 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:02.323606    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.754651574 podStartE2EDuration="5.323583743s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.568932169s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:38:02.322822701 +0000 UTC m=+3547.794662571" watchObservedRunningTime="2026-09-17 16:38:02.323583743 +0000 UTC m=+3547.795423591"
Sep 17 16:38:02.346827 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:02.346787    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:38:02.483497 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:02.483460    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:38:05.820205 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:05.820143    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5" podUID="509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 16:38:14.307874 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:14.307833    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:38:15.830960 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:15.830921    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:38:15.839080 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:15.839050    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:38:30.280817 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:30.280787    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"
Sep 17 16:38:40.031692 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:40.031643    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:38:44.298127 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:44.298097    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5_509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30/main/0.log"
Sep 17 16:38:45.392290 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:45.392238    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"]
Sep 17 16:38:45.397419 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:38:45.397374    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:00.416689 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:00.416638    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5" podUID="509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" containerName="main" containerID="cri-o://4ac8943e63125063ee35eb81bf8deed4361e121961e3819b4c3f8ced7539a7b9" gracePeriod=45
Sep 17 16:39:00.831719 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:00.831621    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5" podUID="509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 16:39:01.261731 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.261703    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:39:01.431684 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.431650    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") pod \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") "
Sep 17 16:39:01.432144 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.431710    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chvlf\" (UniqueName: \"kubernetes.io/projected/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-kube-api-access-chvlf\") pod \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") "
Sep 17 16:39:01.432144 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.431766    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-home\") pod \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") "
Sep 17 16:39:01.432144 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.431800    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-model-cache\") pod \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") "
Sep 17 16:39:01.432144 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.431837    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-tmp-dir\") pod \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") "
Sep 17 16:39:01.432144 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.431867    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-dshm\") pod \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") "
Sep 17 16:39:01.432144 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.431910    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-tls-certs\") pod \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\" (UID: \"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30\") "
Sep 17 16:39:01.432522 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.432224    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-home" (OuterVolumeSpecName: "home") pod "509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" (UID: "509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:39:01.432522 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.432395    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-model-cache" (OuterVolumeSpecName: "model-cache") pod "509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" (UID: "509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:39:01.434235 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.434204    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" (UID: "509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:39:01.434372 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.434215    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-kube-api-access-chvlf" (OuterVolumeSpecName: "kube-api-access-chvlf") pod "509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" (UID: "509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30"). InnerVolumeSpecName "kube-api-access-chvlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:39:01.434641 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.434615    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-dshm" (OuterVolumeSpecName: "dshm") pod "509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" (UID: "509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:39:01.440024 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.440005    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706" (OuterVolumeSpecName: "kserve-pvc-source") pod "509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" (UID: "509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30"). InnerVolumeSpecName "pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 16:39:01.445020 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.444992    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" (UID: "509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:39:01.533362 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.533311    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:39:01.533573 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.533367    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:39:01.533573 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.533386    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:39:01.533573 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.533401    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:39:01.533573 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.533416    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:39:01.533573 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.533490    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") on node \"ip-10-0-129-37.ec2.internal\" "
Sep 17 16:39:01.533573 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.533510    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chvlf\" (UniqueName: \"kubernetes.io/projected/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30-kube-api-access-chvlf\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:39:01.562369 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.562344    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706") on node "ip-10-0-129-37.ec2.internal"
Sep 17 16:39:01.634271 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.634238    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-7057d14c-4e1d-4cef-a2a8-c44bd04aa180\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0797ea84d037b0706\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"csi-6891c468deb24d70da76a392ac947aee09200c4f0a2c290fab5668a762e57dfe\""
Sep 17 16:39:01.656086 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.656052    2574 generic.go:356] "Generic (PLEG): container finished" podID="509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" containerID="4ac8943e63125063ee35eb81bf8deed4361e121961e3819b4c3f8ced7539a7b9" exitCode=0
Sep 17 16:39:01.656222 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.656101    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"
Sep 17 16:39:01.656222 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.656138    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5" event={"ID":"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30","Type":"ContainerDied","Data":"4ac8943e63125063ee35eb81bf8deed4361e121961e3819b4c3f8ced7539a7b9"}
Sep 17 16:39:01.656222 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.656196    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5" event={"ID":"509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30","Type":"ContainerDied","Data":"64700f0fccc75a09680dd486c1bd919312ffd9dada7a27ec80e614fce1d231fe"}
Sep 17 16:39:01.656222 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.656218    2574 scope.go:117] "RemoveContainer" containerID="4ac8943e63125063ee35eb81bf8deed4361e121961e3819b4c3f8ced7539a7b9"
Sep 17 16:39:01.665404 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.665383    2574 scope.go:117] "RemoveContainer" containerID="4ac8943e63125063ee35eb81bf8deed4361e121961e3819b4c3f8ced7539a7b9"
Sep 17 16:39:01.665701 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:39:01.665682    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac8943e63125063ee35eb81bf8deed4361e121961e3819b4c3f8ced7539a7b9\": container with ID starting with 4ac8943e63125063ee35eb81bf8deed4361e121961e3819b4c3f8ced7539a7b9 not found: ID does not exist" containerID="4ac8943e63125063ee35eb81bf8deed4361e121961e3819b4c3f8ced7539a7b9"
Sep 17 16:39:01.665771 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.665720    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac8943e63125063ee35eb81bf8deed4361e121961e3819b4c3f8ced7539a7b9"} err="failed to get container status \"4ac8943e63125063ee35eb81bf8deed4361e121961e3819b4c3f8ced7539a7b9\": rpc error: code = NotFound desc = could not find container \"4ac8943e63125063ee35eb81bf8deed4361e121961e3819b4c3f8ced7539a7b9\": container with ID starting with 4ac8943e63125063ee35eb81bf8deed4361e121961e3819b4c3f8ced7539a7b9 not found: ID does not exist"
Sep 17 16:39:01.680807 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.680766    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"]
Sep 17 16:39:01.683424 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:01.683377    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7k7t5"]
Sep 17 16:39:03.050296 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:03.050258    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" path="/var/lib/kubelet/pods/509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30/volumes"
Sep 17 16:39:05.596961 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:05.596929    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:05.693952 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:05.693916    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:10.573214 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:10.573164    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"]
Sep 17 16:39:10.575110 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:10.575078    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-whghw"]
Sep 17 16:39:11.050029 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:11.049988    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca3eb25d-38dc-4768-90e9-5b86bc60133d" path="/var/lib/kubelet/pods/ca3eb25d-38dc-4768-90e9-5b86bc60133d/volumes"
Sep 17 16:39:11.083179 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:11.083144    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:11.181610 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:11.181573    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:11.953431 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:11.953373    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:13.034136 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:13.034098    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:13.253886 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:13.253849    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:17.106612 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:17.106569    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:18.376690 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:18.376644    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:18.474176 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:18.474129    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:18.629273 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:18.629174    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:18.703186 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:18.703148    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:43.195631 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:43.195594    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"]
Sep 17 16:39:43.197956 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:43.197007    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:43.269929 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:43.269894    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:39:50.279975 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:50.279936    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4" podUID="43e7dd49-495c-4ea0-9a26-fab1c604f712" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:39:58.215817 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:58.215774    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4" podUID="43e7dd49-495c-4ea0-9a26-fab1c604f712" containerName="main" containerID="cri-o://0b8561d6f6f55ac71e97d9b77fb9afa5aa25f4ba460833536beff0037f4566ec" gracePeriod=45
Sep 17 16:39:59.965761 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:39:59.965722    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 16:40:00.280164 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:00.280075    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4" podUID="43e7dd49-495c-4ea0-9a26-fab1c604f712" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:40:10.279766 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:10.279724    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4" podUID="43e7dd49-495c-4ea0-9a26-fab1c604f712" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:40:10.280349 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:10.279839    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"
Sep 17 16:40:13.966764 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:13.966730    2574 scope.go:117] "RemoveContainer" containerID="d3c5c34fd6fdb47fbf9af70744fbf32ebbf00b3d87c6b880368d1092817c326c"
Sep 17 16:40:14.987670 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:14.987621    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="90a92990-f687-4d47-b8a4-9a09d76d2d94" containerName="main" containerID="cri-o://3e3aa068db87d756459b2ec265920b0d0f36d43dc09ae786bf179a4950f75006" gracePeriod=45
Sep 17 16:40:15.732930 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.732902    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:40:15.778014 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.777933    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-home\") pod \"90a92990-f687-4d47-b8a4-9a09d76d2d94\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") "
Sep 17 16:40:15.778014 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.777977    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-kserve-provision-location\") pod \"90a92990-f687-4d47-b8a4-9a09d76d2d94\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") "
Sep 17 16:40:15.778014 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.778009    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-model-cache\") pod \"90a92990-f687-4d47-b8a4-9a09d76d2d94\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") "
Sep 17 16:40:15.778311 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.778027    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a92990-f687-4d47-b8a4-9a09d76d2d94-cabundle-cert\") pod \"90a92990-f687-4d47-b8a4-9a09d76d2d94\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") "
Sep 17 16:40:15.778311 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.778052    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-dshm\") pod \"90a92990-f687-4d47-b8a4-9a09d76d2d94\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") "
Sep 17 16:40:15.778311 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.778073    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxjw4\" (UniqueName: \"kubernetes.io/projected/90a92990-f687-4d47-b8a4-9a09d76d2d94-kube-api-access-dxjw4\") pod \"90a92990-f687-4d47-b8a4-9a09d76d2d94\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") "
Sep 17 16:40:15.778311 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.778145    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-tmp-dir\") pod \"90a92990-f687-4d47-b8a4-9a09d76d2d94\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") "
Sep 17 16:40:15.778311 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.778169    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a92990-f687-4d47-b8a4-9a09d76d2d94-tls-certs\") pod \"90a92990-f687-4d47-b8a4-9a09d76d2d94\" (UID: \"90a92990-f687-4d47-b8a4-9a09d76d2d94\") "
Sep 17 16:40:15.778311 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.778296    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-model-cache" (OuterVolumeSpecName: "model-cache") pod "90a92990-f687-4d47-b8a4-9a09d76d2d94" (UID: "90a92990-f687-4d47-b8a4-9a09d76d2d94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:40:15.778655 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.778409    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a92990-f687-4d47-b8a4-9a09d76d2d94-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "90a92990-f687-4d47-b8a4-9a09d76d2d94" (UID: "90a92990-f687-4d47-b8a4-9a09d76d2d94"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:40:15.778655 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.778515    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:40:15.778655 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.778533    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a92990-f687-4d47-b8a4-9a09d76d2d94-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:40:15.779087 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.778858    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-home" (OuterVolumeSpecName: "home") pod "90a92990-f687-4d47-b8a4-9a09d76d2d94" (UID: "90a92990-f687-4d47-b8a4-9a09d76d2d94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:40:15.781267 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.781149    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a92990-f687-4d47-b8a4-9a09d76d2d94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90a92990-f687-4d47-b8a4-9a09d76d2d94" (UID: "90a92990-f687-4d47-b8a4-9a09d76d2d94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:40:15.781267 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.781165    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a92990-f687-4d47-b8a4-9a09d76d2d94-kube-api-access-dxjw4" (OuterVolumeSpecName: "kube-api-access-dxjw4") pod "90a92990-f687-4d47-b8a4-9a09d76d2d94" (UID: "90a92990-f687-4d47-b8a4-9a09d76d2d94"). InnerVolumeSpecName "kube-api-access-dxjw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:40:15.781267 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.781231    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-dshm" (OuterVolumeSpecName: "dshm") pod "90a92990-f687-4d47-b8a4-9a09d76d2d94" (UID: "90a92990-f687-4d47-b8a4-9a09d76d2d94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:40:15.793692 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.793661    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90a92990-f687-4d47-b8a4-9a09d76d2d94" (UID: "90a92990-f687-4d47-b8a4-9a09d76d2d94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:40:15.800590 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.800553    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90a92990-f687-4d47-b8a4-9a09d76d2d94" (UID: "90a92990-f687-4d47-b8a4-9a09d76d2d94"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:40:15.878939 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.878900    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjw4\" (UniqueName: \"kubernetes.io/projected/90a92990-f687-4d47-b8a4-9a09d76d2d94-kube-api-access-dxjw4\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:40:15.879112 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.878955    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:40:15.879112 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.878965    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a92990-f687-4d47-b8a4-9a09d76d2d94-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:40:15.879112 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.878974    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:40:15.879112 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.878983    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:40:15.879112 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:15.878991    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a92990-f687-4d47-b8a4-9a09d76d2d94-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:40:16.033604 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:16.033512    2574 generic.go:356] "Generic (PLEG): container finished" podID="90a92990-f687-4d47-b8a4-9a09d76d2d94" containerID="3e3aa068db87d756459b2ec265920b0d0f36d43dc09ae786bf179a4950f75006" exitCode=0
Sep 17 16:40:16.033604 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:16.033563    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 16:40:16.034007 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:16.033606    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"90a92990-f687-4d47-b8a4-9a09d76d2d94","Type":"ContainerDied","Data":"3e3aa068db87d756459b2ec265920b0d0f36d43dc09ae786bf179a4950f75006"}
Sep 17 16:40:16.034007 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:16.033668    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"90a92990-f687-4d47-b8a4-9a09d76d2d94","Type":"ContainerDied","Data":"6fe1d2b7c8662413273936a33159ae41014b2e388071694163297ffa747d239e"}
Sep 17 16:40:16.034007 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:16.033699    2574 scope.go:117] "RemoveContainer" containerID="3e3aa068db87d756459b2ec265920b0d0f36d43dc09ae786bf179a4950f75006"
Sep 17 16:40:16.043248 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:16.043229    2574 scope.go:117] "RemoveContainer" containerID="cc054125eb7e730cb9b7734019436d2c915ff1e89bd9aa951f2d5fd16d95b70d"
Sep 17 16:40:16.051051 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:16.051030    2574 scope.go:117] "RemoveContainer" containerID="3e3aa068db87d756459b2ec265920b0d0f36d43dc09ae786bf179a4950f75006"
Sep 17 16:40:16.051315 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:40:16.051293    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3aa068db87d756459b2ec265920b0d0f36d43dc09ae786bf179a4950f75006\": container with ID starting with 3e3aa068db87d756459b2ec265920b0d0f36d43dc09ae786bf179a4950f75006 not found: ID does not exist" containerID="3e3aa068db87d756459b2ec265920b0d0f36d43dc09ae786bf179a4950f75006"
Sep 17 16:40:16.051400 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:16.051322    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3aa068db87d756459b2ec265920b0d0f36d43dc09ae786bf179a4950f75006"} err="failed to get container status \"3e3aa068db87d756459b2ec265920b0d0f36d43dc09ae786bf179a4950f75006\": rpc error: code = NotFound desc = could not find container \"3e3aa068db87d756459b2ec265920b0d0f36d43dc09ae786bf179a4950f75006\": container with ID starting with 3e3aa068db87d756459b2ec265920b0d0f36d43dc09ae786bf179a4950f75006 not found: ID does not exist"
Sep 17 16:40:16.051400 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:16.051339    2574 scope.go:117] "RemoveContainer" containerID="cc054125eb7e730cb9b7734019436d2c915ff1e89bd9aa951f2d5fd16d95b70d"
Sep 17 16:40:16.051618 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:40:16.051599    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc054125eb7e730cb9b7734019436d2c915ff1e89bd9aa951f2d5fd16d95b70d\": container with ID starting with cc054125eb7e730cb9b7734019436d2c915ff1e89bd9aa951f2d5fd16d95b70d not found: ID does not exist" containerID="cc054125eb7e730cb9b7734019436d2c915ff1e89bd9aa951f2d5fd16d95b70d"
Sep 17 16:40:16.051659 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:16.051623    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc054125eb7e730cb9b7734019436d2c915ff1e89bd9aa951f2d5fd16d95b70d"} err="failed to get container status \"cc054125eb7e730cb9b7734019436d2c915ff1e89bd9aa951f2d5fd16d95b70d\": rpc error: code = NotFound desc = could not find container \"cc054125eb7e730cb9b7734019436d2c915ff1e89bd9aa951f2d5fd16d95b70d\": container with ID starting with cc054125eb7e730cb9b7734019436d2c915ff1e89bd9aa951f2d5fd16d95b70d not found: ID does not exist"
Sep 17 16:40:16.057061 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:16.057035    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 16:40:16.061963 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:16.061942    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 16:40:17.048740 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:17.048700    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90a92990-f687-4d47-b8a4-9a09d76d2d94" path="/var/lib/kubelet/pods/90a92990-f687-4d47-b8a4-9a09d76d2d94/volumes"
Sep 17 16:40:20.280694 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:20.280644    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4" podUID="43e7dd49-495c-4ea0-9a26-fab1c604f712" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:40:28.474732 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:28.474708    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"
Sep 17 16:40:28.585297 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:28.585211    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5884\" (UniqueName: \"kubernetes.io/projected/43e7dd49-495c-4ea0-9a26-fab1c604f712-kube-api-access-s5884\") pod \"43e7dd49-495c-4ea0-9a26-fab1c604f712\" (UID: \"43e7dd49-495c-4ea0-9a26-fab1c604f712\") "
Sep 17 16:40:28.585297 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:28.585254    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e7dd49-495c-4ea0-9a26-fab1c604f712-tls-certs\") pod \"43e7dd49-495c-4ea0-9a26-fab1c604f712\" (UID: \"43e7dd49-495c-4ea0-9a26-fab1c604f712\") "
Sep 17 16:40:28.587483 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:28.587455    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e7dd49-495c-4ea0-9a26-fab1c604f712-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43e7dd49-495c-4ea0-9a26-fab1c604f712" (UID: "43e7dd49-495c-4ea0-9a26-fab1c604f712"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:40:28.587553 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:28.587501    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e7dd49-495c-4ea0-9a26-fab1c604f712-kube-api-access-s5884" (OuterVolumeSpecName: "kube-api-access-s5884") pod "43e7dd49-495c-4ea0-9a26-fab1c604f712" (UID: "43e7dd49-495c-4ea0-9a26-fab1c604f712"). InnerVolumeSpecName "kube-api-access-s5884". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:40:28.686225 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:28.686183    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5884\" (UniqueName: \"kubernetes.io/projected/43e7dd49-495c-4ea0-9a26-fab1c604f712-kube-api-access-s5884\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:40:28.686225 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:28.686218    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e7dd49-495c-4ea0-9a26-fab1c604f712-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:40:29.094607 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:29.094573    2574 generic.go:356] "Generic (PLEG): container finished" podID="43e7dd49-495c-4ea0-9a26-fab1c604f712" containerID="0b8561d6f6f55ac71e97d9b77fb9afa5aa25f4ba460833536beff0037f4566ec" exitCode=0
Sep 17 16:40:29.094765 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:29.094625    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"
Sep 17 16:40:29.094765 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:29.094639    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4" event={"ID":"43e7dd49-495c-4ea0-9a26-fab1c604f712","Type":"ContainerDied","Data":"0b8561d6f6f55ac71e97d9b77fb9afa5aa25f4ba460833536beff0037f4566ec"}
Sep 17 16:40:29.094765 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:29.094681    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4" event={"ID":"43e7dd49-495c-4ea0-9a26-fab1c604f712","Type":"ContainerDied","Data":"8bdc7ce7d3c12286f0dd1a859cb32cdedaa91690e179704c37b9aea51ef56a5a"}
Sep 17 16:40:29.094765 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:29.094703    2574 scope.go:117] "RemoveContainer" containerID="0b8561d6f6f55ac71e97d9b77fb9afa5aa25f4ba460833536beff0037f4566ec"
Sep 17 16:40:29.103288 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:29.103268    2574 scope.go:117] "RemoveContainer" containerID="0b8561d6f6f55ac71e97d9b77fb9afa5aa25f4ba460833536beff0037f4566ec"
Sep 17 16:40:29.103600 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:40:29.103578    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8561d6f6f55ac71e97d9b77fb9afa5aa25f4ba460833536beff0037f4566ec\": container with ID starting with 0b8561d6f6f55ac71e97d9b77fb9afa5aa25f4ba460833536beff0037f4566ec not found: ID does not exist" containerID="0b8561d6f6f55ac71e97d9b77fb9afa5aa25f4ba460833536beff0037f4566ec"
Sep 17 16:40:29.103654 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:29.103608    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8561d6f6f55ac71e97d9b77fb9afa5aa25f4ba460833536beff0037f4566ec"} err="failed to get container status \"0b8561d6f6f55ac71e97d9b77fb9afa5aa25f4ba460833536beff0037f4566ec\": rpc error: code = NotFound desc = could not find container \"0b8561d6f6f55ac71e97d9b77fb9afa5aa25f4ba460833536beff0037f4566ec\": container with ID starting with 0b8561d6f6f55ac71e97d9b77fb9afa5aa25f4ba460833536beff0037f4566ec not found: ID does not exist"
Sep 17 16:40:29.114506 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:29.114482    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"]
Sep 17 16:40:29.121515 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:29.121492    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehm7s4"]
Sep 17 16:40:31.048607 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:31.048574    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43e7dd49-495c-4ea0-9a26-fab1c604f712" path="/var/lib/kubelet/pods/43e7dd49-495c-4ea0-9a26-fab1c604f712/volumes"
Sep 17 16:40:33.335457 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:33.335406    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:40:33.433115 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:33.433063    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:40:40.161170 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:40.161127    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:40:47.352275 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:47.352230    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:40:47.456144 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:47.456110    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:40:49.613514 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.613480    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"]
Sep 17 16:40:49.613904 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.613873    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" containerName="main"
Sep 17 16:40:49.613904 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.613885    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" containerName="main"
Sep 17 16:40:49.613904 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.613895    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e7dd49-495c-4ea0-9a26-fab1c604f712" containerName="main"
Sep 17 16:40:49.613904 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.613900    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e7dd49-495c-4ea0-9a26-fab1c604f712" containerName="main"
Sep 17 16:40:49.614033 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.613908    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90a92990-f687-4d47-b8a4-9a09d76d2d94" containerName="storage-initializer"
Sep 17 16:40:49.614033 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.613914    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="90a92990-f687-4d47-b8a4-9a09d76d2d94" containerName="storage-initializer"
Sep 17 16:40:49.614033 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.613929    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90a92990-f687-4d47-b8a4-9a09d76d2d94" containerName="main"
Sep 17 16:40:49.614033 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.613934    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="90a92990-f687-4d47-b8a4-9a09d76d2d94" containerName="main"
Sep 17 16:40:49.614033 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.613981    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e7dd49-495c-4ea0-9a26-fab1c604f712" containerName="main"
Sep 17 16:40:49.614033 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.613987    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="90a92990-f687-4d47-b8a4-9a09d76d2d94" containerName="main"
Sep 17 16:40:49.614033 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.613996    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="509a3b6f-24b8-4bfd-a6eb-e5c973c4ef30" containerName="main"
Sep 17 16:40:49.616853 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.616832    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.619223 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.619203    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 16:40:49.620015 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.619992    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-m45gc\""
Sep 17 16:40:49.620100 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.620027    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 16:40:49.620100 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.619992    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 16:40:49.630754 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.630728    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"]
Sep 17 16:40:49.638267 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.638231    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:40:49.761205 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.761169    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.761386 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.761257    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.761386 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.761289    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.761386 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.761312    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggr6w\" (UniqueName: \"kubernetes.io/projected/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-kube-api-access-ggr6w\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.761386 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.761363    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.761557 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.761388    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.862575 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.862537    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.862767 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.862591    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.862767 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.862621    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggr6w\" (UniqueName: \"kubernetes.io/projected/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-kube-api-access-ggr6w\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.862767 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.862647    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.862767 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.862672    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.862767 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.862697    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.863064 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.863037    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.863210 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.863187    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.863383 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.863344    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.865615 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.865541    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.866101 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.866081    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.874696 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.874663    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"]
Sep 17 16:40:49.877491 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.877461    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggr6w\" (UniqueName: \"kubernetes.io/projected/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-kube-api-access-ggr6w\") pod \"scheduler-inline-config-test-kserve-8655bc757-dftks\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.878661 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.878641    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"
Sep 17 16:40:49.881022 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.881003    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-4dk2f\""
Sep 17 16:40:49.887159 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.887131    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"]
Sep 17 16:40:49.927118 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.927086    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:49.947203 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.947157    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:40:49.963484 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.963429    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2953dc8-3133-4044-8e96-93d105972b89-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb92699\" (UID: \"c2953dc8-3133-4044-8e96-93d105972b89\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"
Sep 17 16:40:49.963641 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:49.963516    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5w7p\" (UniqueName: \"kubernetes.io/projected/c2953dc8-3133-4044-8e96-93d105972b89-kube-api-access-l5w7p\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb92699\" (UID: \"c2953dc8-3133-4044-8e96-93d105972b89\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"
Sep 17 16:40:50.064806 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:50.064765    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2953dc8-3133-4044-8e96-93d105972b89-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb92699\" (UID: \"c2953dc8-3133-4044-8e96-93d105972b89\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"
Sep 17 16:40:50.064806 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:50.064805    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5w7p\" (UniqueName: \"kubernetes.io/projected/c2953dc8-3133-4044-8e96-93d105972b89-kube-api-access-l5w7p\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb92699\" (UID: \"c2953dc8-3133-4044-8e96-93d105972b89\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"
Sep 17 16:40:50.067511 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:50.067486    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2953dc8-3133-4044-8e96-93d105972b89-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb92699\" (UID: \"c2953dc8-3133-4044-8e96-93d105972b89\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"
Sep 17 16:40:50.072971 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:50.072943    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5w7p\" (UniqueName: \"kubernetes.io/projected/c2953dc8-3133-4044-8e96-93d105972b89-kube-api-access-l5w7p\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb92699\" (UID: \"c2953dc8-3133-4044-8e96-93d105972b89\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"
Sep 17 16:40:50.203798 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:50.203710    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"
Sep 17 16:40:50.207656 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:50.207626    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"]
Sep 17 16:40:50.211082 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:40:50.211036    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9bc3bb_b4dd_4034_8286_1b17779c0fed.slice/crio-0fd216b6c34a91b00f31c67b1168efacaa10c4c1639ab82b9119f7b3a5f6dc86 WatchSource:0}: Error finding container 0fd216b6c34a91b00f31c67b1168efacaa10c4c1639ab82b9119f7b3a5f6dc86: Status 404 returned error can't find the container with id 0fd216b6c34a91b00f31c67b1168efacaa10c4c1639ab82b9119f7b3a5f6dc86
Sep 17 16:40:50.484803 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:50.484775    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"]
Sep 17 16:40:50.487410 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:40:50.487384    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2953dc8_3133_4044_8e96_93d105972b89.slice/crio-ffcf793a43f3cd579d88d442ecc1923ff38962f1845fb2a6a2bc19944e319d45 WatchSource:0}: Error finding container ffcf793a43f3cd579d88d442ecc1923ff38962f1845fb2a6a2bc19944e319d45: Status 404 returned error can't find the container with id ffcf793a43f3cd579d88d442ecc1923ff38962f1845fb2a6a2bc19944e319d45
Sep 17 16:40:51.203365 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:51.203328    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks" event={"ID":"7d9bc3bb-b4dd-4034-8286-1b17779c0fed","Type":"ContainerStarted","Data":"d2e33a442d254da8630004d7bcf95e76cb62cbe1c36a2f7b4729cc0fab72330f"}
Sep 17 16:40:51.203864 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:51.203383    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks" event={"ID":"7d9bc3bb-b4dd-4034-8286-1b17779c0fed","Type":"ContainerStarted","Data":"0fd216b6c34a91b00f31c67b1168efacaa10c4c1639ab82b9119f7b3a5f6dc86"}
Sep 17 16:40:51.204765 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:51.204739    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699" event={"ID":"c2953dc8-3133-4044-8e96-93d105972b89","Type":"ContainerStarted","Data":"7c78c0648cdeb6459b5f43092ffa7754baa1a88803917cc0ba7ffa422dc7f84f"}
Sep 17 16:40:51.204869 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:51.204789    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699" event={"ID":"c2953dc8-3133-4044-8e96-93d105972b89","Type":"ContainerStarted","Data":"ffcf793a43f3cd579d88d442ecc1923ff38962f1845fb2a6a2bc19944e319d45"}
Sep 17 16:40:51.204869 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:51.204822    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"
Sep 17 16:40:51.224841 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:51.224790    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks" podStartSLOduration=2.224773778 podStartE2EDuration="2.224773778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:40:51.223076947 +0000 UTC m=+3716.694916792" watchObservedRunningTime="2026-09-17 16:40:51.224773778 +0000 UTC m=+3716.696613624"
Sep 17 16:40:51.239842 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:51.239787    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699" podStartSLOduration=2.239768631 podStartE2EDuration="2.239768631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:40:51.238932396 +0000 UTC m=+3716.710772243" watchObservedRunningTime="2026-09-17 16:40:51.239768631 +0000 UTC m=+3716.711608479"
Sep 17 16:40:51.928233 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:51.928189    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:59.702063 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:59.702017    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:40:59.927509 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:59.927461    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:40:59.940418 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:40:59.940384    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:41:00.266173 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:00.266143    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:41:17.856810 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:17.856752    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:41:17.956355 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:17.956320    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:41:22.212811 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:22.212776    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"
Sep 17 16:41:24.112026 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.111980    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:41:24.117213 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.117179    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"]
Sep 17 16:41:24.120681 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.120650    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"]
Sep 17 16:41:24.127580 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:41:24.127540    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:41:24.127580 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:41:24.127580 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:41:24.127580 ip-10-0-129-37 kubenswrapper[2574]:  > containerID="d2e33a442d254da8630004d7bcf95e76cb62cbe1c36a2f7b4729cc0fab72330f" cmd=["/bin/sleep","15"]
Sep 17 16:41:24.127817 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.127606    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks" message="" err=<
Sep 17 16:41:24.127817 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:41:24.127817 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:41:24.127817 ip-10-0-129-37 kubenswrapper[2574]:  >
Sep 17 16:41:24.127817 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:41:24.127624    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:41:24.127817 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:41:24.127817 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:41:24.127817 ip-10-0-129-37 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks" podUID="7d9bc3bb-b4dd-4034-8286-1b17779c0fed" containerName="main" containerID="cri-o://d2e33a442d254da8630004d7bcf95e76cb62cbe1c36a2f7b4729cc0fab72330f"
Sep 17 16:41:24.127817 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.127671    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks" podUID="7d9bc3bb-b4dd-4034-8286-1b17779c0fed" containerName="main" containerID="cri-o://d2e33a442d254da8630004d7bcf95e76cb62cbe1c36a2f7b4729cc0fab72330f" gracePeriod=60
Sep 17 16:41:24.181524 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.181481    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:41:24.255193 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.255140    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks" podUID="7d9bc3bb-b4dd-4034-8286-1b17779c0fed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 16:41:24.379557 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.379479    2574 generic.go:356] "Generic (PLEG): container finished" podID="7d9bc3bb-b4dd-4034-8286-1b17779c0fed" containerID="d2e33a442d254da8630004d7bcf95e76cb62cbe1c36a2f7b4729cc0fab72330f" exitCode=0
Sep 17 16:41:24.379557 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.379518    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks" event={"ID":"7d9bc3bb-b4dd-4034-8286-1b17779c0fed","Type":"ContainerDied","Data":"d2e33a442d254da8630004d7bcf95e76cb62cbe1c36a2f7b4729cc0fab72330f"}
Sep 17 16:41:24.875520 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.875479    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:41:24.970278 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.970245    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-model-cache\") pod \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") "
Sep 17 16:41:24.970517 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.970291    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggr6w\" (UniqueName: \"kubernetes.io/projected/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-kube-api-access-ggr6w\") pod \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") "
Sep 17 16:41:24.970517 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.970384    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-tls-certs\") pod \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") "
Sep 17 16:41:24.970517 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.970476    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-dshm\") pod \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") "
Sep 17 16:41:24.970517 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.970516    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-tmp-dir\") pod \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") "
Sep 17 16:41:24.970751 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.970588    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-home\") pod \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\" (UID: \"7d9bc3bb-b4dd-4034-8286-1b17779c0fed\") "
Sep 17 16:41:24.970751 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.970583    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d9bc3bb-b4dd-4034-8286-1b17779c0fed" (UID: "7d9bc3bb-b4dd-4034-8286-1b17779c0fed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:41:24.970852 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.970808    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d9bc3bb-b4dd-4034-8286-1b17779c0fed" (UID: "7d9bc3bb-b4dd-4034-8286-1b17779c0fed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:41:24.970852 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.970831    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:41:24.970933 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.970863    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-home" (OuterVolumeSpecName: "home") pod "7d9bc3bb-b4dd-4034-8286-1b17779c0fed" (UID: "7d9bc3bb-b4dd-4034-8286-1b17779c0fed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:41:24.972797 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.972770    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-dshm" (OuterVolumeSpecName: "dshm") pod "7d9bc3bb-b4dd-4034-8286-1b17779c0fed" (UID: "7d9bc3bb-b4dd-4034-8286-1b17779c0fed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:41:24.972797 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.972782    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d9bc3bb-b4dd-4034-8286-1b17779c0fed" (UID: "7d9bc3bb-b4dd-4034-8286-1b17779c0fed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:41:24.972936 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:24.972800    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-kube-api-access-ggr6w" (OuterVolumeSpecName: "kube-api-access-ggr6w") pod "7d9bc3bb-b4dd-4034-8286-1b17779c0fed" (UID: "7d9bc3bb-b4dd-4034-8286-1b17779c0fed"). InnerVolumeSpecName "kube-api-access-ggr6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:41:25.071963 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:25.071919    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:41:25.071963 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:25.071954    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:41:25.071963 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:25.071967    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:41:25.072196 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:25.071978    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:41:25.072196 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:25.071990    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggr6w\" (UniqueName: \"kubernetes.io/projected/7d9bc3bb-b4dd-4034-8286-1b17779c0fed-kube-api-access-ggr6w\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:41:25.385245 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:25.385213    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"
Sep 17 16:41:25.385677 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:25.385221    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks" event={"ID":"7d9bc3bb-b4dd-4034-8286-1b17779c0fed","Type":"ContainerDied","Data":"0fd216b6c34a91b00f31c67b1168efacaa10c4c1639ab82b9119f7b3a5f6dc86"}
Sep 17 16:41:25.386134 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:25.385359    2574 scope.go:117] "RemoveContainer" containerID="d2e33a442d254da8630004d7bcf95e76cb62cbe1c36a2f7b4729cc0fab72330f"
Sep 17 16:41:25.404083 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:25.404057    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"]
Sep 17 16:41:25.407206 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:25.407181    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dftks"]
Sep 17 16:41:27.049158 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:27.049122    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d9bc3bb-b4dd-4034-8286-1b17779c0fed" path="/var/lib/kubelet/pods/7d9bc3bb-b4dd-4034-8286-1b17779c0fed/volumes"
Sep 17 16:41:32.213376 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:32.213326    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699" podUID="c2953dc8-3133-4044-8e96-93d105972b89" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:41:39.147634 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:39.147566    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699" podUID="c2953dc8-3133-4044-8e96-93d105972b89" containerName="main" containerID="cri-o://7c78c0648cdeb6459b5f43092ffa7754baa1a88803917cc0ba7ffa422dc7f84f" gracePeriod=45
Sep 17 16:41:42.213353 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:42.213302    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699" podUID="c2953dc8-3133-4044-8e96-93d105972b89" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:41:52.213837 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:52.213756    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699" podUID="c2953dc8-3133-4044-8e96-93d105972b89" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:41:52.214220 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:41:52.213871    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"
Sep 17 16:42:02.213120 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:02.213080    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699" podUID="c2953dc8-3133-4044-8e96-93d105972b89" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:42:07.947513 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:07.947471    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:08.043148 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:08.043107    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:09.406905 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.406844    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"
Sep 17 16:42:09.453407 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.453376    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2953dc8-3133-4044-8e96-93d105972b89-tls-certs\") pod \"c2953dc8-3133-4044-8e96-93d105972b89\" (UID: \"c2953dc8-3133-4044-8e96-93d105972b89\") "
Sep 17 16:42:09.453588 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.453429    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5w7p\" (UniqueName: \"kubernetes.io/projected/c2953dc8-3133-4044-8e96-93d105972b89-kube-api-access-l5w7p\") pod \"c2953dc8-3133-4044-8e96-93d105972b89\" (UID: \"c2953dc8-3133-4044-8e96-93d105972b89\") "
Sep 17 16:42:09.455573 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.455541    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2953dc8-3133-4044-8e96-93d105972b89-kube-api-access-l5w7p" (OuterVolumeSpecName: "kube-api-access-l5w7p") pod "c2953dc8-3133-4044-8e96-93d105972b89" (UID: "c2953dc8-3133-4044-8e96-93d105972b89"). InnerVolumeSpecName "kube-api-access-l5w7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:42:09.455692 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.455555    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2953dc8-3133-4044-8e96-93d105972b89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2953dc8-3133-4044-8e96-93d105972b89" (UID: "c2953dc8-3133-4044-8e96-93d105972b89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:42:09.554734 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.554697    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2953dc8-3133-4044-8e96-93d105972b89-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:42:09.554734 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.554729    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5w7p\" (UniqueName: \"kubernetes.io/projected/c2953dc8-3133-4044-8e96-93d105972b89-kube-api-access-l5w7p\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:42:09.601333 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.601297    2574 generic.go:356] "Generic (PLEG): container finished" podID="c2953dc8-3133-4044-8e96-93d105972b89" containerID="7c78c0648cdeb6459b5f43092ffa7754baa1a88803917cc0ba7ffa422dc7f84f" exitCode=0
Sep 17 16:42:09.601548 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.601348    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"
Sep 17 16:42:09.601548 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.601383    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699" event={"ID":"c2953dc8-3133-4044-8e96-93d105972b89","Type":"ContainerDied","Data":"7c78c0648cdeb6459b5f43092ffa7754baa1a88803917cc0ba7ffa422dc7f84f"}
Sep 17 16:42:09.601548 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.601470    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699" event={"ID":"c2953dc8-3133-4044-8e96-93d105972b89","Type":"ContainerDied","Data":"ffcf793a43f3cd579d88d442ecc1923ff38962f1845fb2a6a2bc19944e319d45"}
Sep 17 16:42:09.601548 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.601504    2574 scope.go:117] "RemoveContainer" containerID="7c78c0648cdeb6459b5f43092ffa7754baa1a88803917cc0ba7ffa422dc7f84f"
Sep 17 16:42:09.610572 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.610549    2574 scope.go:117] "RemoveContainer" containerID="7c78c0648cdeb6459b5f43092ffa7754baa1a88803917cc0ba7ffa422dc7f84f"
Sep 17 16:42:09.610854 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:42:09.610825    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c78c0648cdeb6459b5f43092ffa7754baa1a88803917cc0ba7ffa422dc7f84f\": container with ID starting with 7c78c0648cdeb6459b5f43092ffa7754baa1a88803917cc0ba7ffa422dc7f84f not found: ID does not exist" containerID="7c78c0648cdeb6459b5f43092ffa7754baa1a88803917cc0ba7ffa422dc7f84f"
Sep 17 16:42:09.610960 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.610862    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c78c0648cdeb6459b5f43092ffa7754baa1a88803917cc0ba7ffa422dc7f84f"} err="failed to get container status \"7c78c0648cdeb6459b5f43092ffa7754baa1a88803917cc0ba7ffa422dc7f84f\": rpc error: code = NotFound desc = could not find container \"7c78c0648cdeb6459b5f43092ffa7754baa1a88803917cc0ba7ffa422dc7f84f\": container with ID starting with 7c78c0648cdeb6459b5f43092ffa7754baa1a88803917cc0ba7ffa422dc7f84f not found: ID does not exist"
Sep 17 16:42:09.623256 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.623225    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"]
Sep 17 16:42:09.627288 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:09.627260    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb92699"]
Sep 17 16:42:11.053974 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:11.053939    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2953dc8-3133-4044-8e96-93d105972b89" path="/var/lib/kubelet/pods/c2953dc8-3133-4044-8e96-93d105972b89/volumes"
Sep 17 16:42:13.505824 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.505776    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:13.626007 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.625969    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:13.673639 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.673601    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"]
Sep 17 16:42:13.674039 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.674023    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2953dc8-3133-4044-8e96-93d105972b89" containerName="main"
Sep 17 16:42:13.674039 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.674039    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2953dc8-3133-4044-8e96-93d105972b89" containerName="main"
Sep 17 16:42:13.674146 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.674058    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9bc3bb-b4dd-4034-8286-1b17779c0fed" containerName="main"
Sep 17 16:42:13.674146 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.674064    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9bc3bb-b4dd-4034-8286-1b17779c0fed" containerName="main"
Sep 17 16:42:13.674146 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.674137    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2953dc8-3133-4044-8e96-93d105972b89" containerName="main"
Sep 17 16:42:13.674247 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.674147    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d9bc3bb-b4dd-4034-8286-1b17779c0fed" containerName="main"
Sep 17 16:42:13.677379 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.677361    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.680855 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.680832    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-ktq5d\""
Sep 17 16:42:13.681137 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.681121    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 16:42:13.681537 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.681519    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 16:42:13.681599 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.681519    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 16:42:13.681995 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.681976    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 16:42:13.682693 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.682677    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:42:13.748939 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.748901    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:13.761606 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.761526    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"]
Sep 17 16:42:13.792007 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.791914    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-cabundle-cert\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.792007 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.791950    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-tmp-dir\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.792007 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.791981    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-model-cache\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.792240 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.792065    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-dshm\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.792240 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.792090    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-home\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.792240 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.792112    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-tls-certs\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.792240 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.792151    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95zwh\" (UniqueName: \"kubernetes.io/projected/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-kube-api-access-95zwh\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.792240 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.792177    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-kserve-provision-location\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.893093 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.893059    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-tmp-dir\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.893093 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.893094    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-model-cache\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.893304 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.893232    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-dshm\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.893304 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.893271    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-home\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.893304 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.893295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-tls-certs\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.893507 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.893358    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95zwh\" (UniqueName: \"kubernetes.io/projected/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-kube-api-access-95zwh\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.893507 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.893396    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-kserve-provision-location\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.893507 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.893498    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-cabundle-cert\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.893682 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.893512    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-tmp-dir\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.893682 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.893535    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-model-cache\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.893791 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.893683    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-home\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.893861 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.893841    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-kserve-provision-location\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.894103 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.894086    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-cabundle-cert\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.895680 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.895659    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-dshm\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.895869 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.895853    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-tls-certs\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.903010 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.902985    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95zwh\" (UniqueName: \"kubernetes.io/projected/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-kube-api-access-95zwh\") pod \"prestop-hook-test-kserve-6989b4f56b-c9xkx\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.982731 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.982692    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"]
Sep 17 16:42:13.987525 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.987500    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"
Sep 17 16:42:13.988554 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.988536    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:13.989695 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.989677    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-vk4ff\""
Sep 17 16:42:13.995519 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:13.995308    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"]
Sep 17 16:42:14.078089 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:14.078057    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:14.094830 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:14.094792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbtzh\" (UniqueName: \"kubernetes.io/projected/2218528e-4c40-4e46-b97f-52f102c1a1e0-kube-api-access-fbtzh\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66\" (UID: \"2218528e-4c40-4e46-b97f-52f102c1a1e0\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"
Sep 17 16:42:14.094996 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:14.094896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2218528e-4c40-4e46-b97f-52f102c1a1e0-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66\" (UID: \"2218528e-4c40-4e46-b97f-52f102c1a1e0\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"
Sep 17 16:42:14.196388 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:14.196347    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2218528e-4c40-4e46-b97f-52f102c1a1e0-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66\" (UID: \"2218528e-4c40-4e46-b97f-52f102c1a1e0\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"
Sep 17 16:42:14.196605 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:14.196409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbtzh\" (UniqueName: \"kubernetes.io/projected/2218528e-4c40-4e46-b97f-52f102c1a1e0-kube-api-access-fbtzh\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66\" (UID: \"2218528e-4c40-4e46-b97f-52f102c1a1e0\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"
Sep 17 16:42:14.199124 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:14.199097    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2218528e-4c40-4e46-b97f-52f102c1a1e0-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66\" (UID: \"2218528e-4c40-4e46-b97f-52f102c1a1e0\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"
Sep 17 16:42:14.205016 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:14.204982    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbtzh\" (UniqueName: \"kubernetes.io/projected/2218528e-4c40-4e46-b97f-52f102c1a1e0-kube-api-access-fbtzh\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66\" (UID: \"2218528e-4c40-4e46-b97f-52f102c1a1e0\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"
Sep 17 16:42:14.273885 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:14.273848    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"]
Sep 17 16:42:14.276213 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:42:14.276182    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c17bbec_7fea_44e3_9202_e2ca21ad9d33.slice/crio-f6b24b8e49209caff9e198789bdc307dedd4aeed37c06db950a81d6cd1f452ea WatchSource:0}: Error finding container f6b24b8e49209caff9e198789bdc307dedd4aeed37c06db950a81d6cd1f452ea: Status 404 returned error can't find the container with id f6b24b8e49209caff9e198789bdc307dedd4aeed37c06db950a81d6cd1f452ea
Sep 17 16:42:14.299886 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:14.299860    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"
Sep 17 16:42:14.576685 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:14.576576    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"]
Sep 17 16:42:14.579326 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:42:14.579291    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2218528e_4c40_4e46_b97f_52f102c1a1e0.slice/crio-1721926d5e983495201f12005b91da3157977ec0b41956687a170b5b270bd107 WatchSource:0}: Error finding container 1721926d5e983495201f12005b91da3157977ec0b41956687a170b5b270bd107: Status 404 returned error can't find the container with id 1721926d5e983495201f12005b91da3157977ec0b41956687a170b5b270bd107
Sep 17 16:42:14.629030 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:14.628990    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" event={"ID":"9c17bbec-7fea-44e3-9202-e2ca21ad9d33","Type":"ContainerStarted","Data":"18340be96d6759c28717ba34fd4b5deaabd2994f150e44eec0229cd980c419b6"}
Sep 17 16:42:14.629207 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:14.629062    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" event={"ID":"9c17bbec-7fea-44e3-9202-e2ca21ad9d33","Type":"ContainerStarted","Data":"f6b24b8e49209caff9e198789bdc307dedd4aeed37c06db950a81d6cd1f452ea"}
Sep 17 16:42:14.630368 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:14.630341    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66" event={"ID":"2218528e-4c40-4e46-b97f-52f102c1a1e0","Type":"ContainerStarted","Data":"1721926d5e983495201f12005b91da3157977ec0b41956687a170b5b270bd107"}
Sep 17 16:42:15.638376 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:15.638328    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66" event={"ID":"2218528e-4c40-4e46-b97f-52f102c1a1e0","Type":"ContainerStarted","Data":"392ea308d3733bbde78ef8cec36dee33342a52bd0e1eb59fb5e334713f0e4b5e"}
Sep 17 16:42:15.638791 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:15.638604    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"
Sep 17 16:42:15.658282 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:15.658233    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66" podStartSLOduration=2.658217909 podStartE2EDuration="2.658217909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:15.656529859 +0000 UTC m=+3801.128369706" watchObservedRunningTime="2026-09-17 16:42:15.658217909 +0000 UTC m=+3801.130057754"
Sep 17 16:42:15.756538 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:15.756506    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:17.650543 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:17.650508    2574 generic.go:356] "Generic (PLEG): container finished" podID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerID="18340be96d6759c28717ba34fd4b5deaabd2994f150e44eec0229cd980c419b6" exitCode=0
Sep 17 16:42:17.650931 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:17.650555    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" event={"ID":"9c17bbec-7fea-44e3-9202-e2ca21ad9d33","Type":"ContainerDied","Data":"18340be96d6759c28717ba34fd4b5deaabd2994f150e44eec0229cd980c419b6"}
Sep 17 16:42:18.657491 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:18.657431    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" event={"ID":"9c17bbec-7fea-44e3-9202-e2ca21ad9d33","Type":"ContainerStarted","Data":"f573471fee28c793ad5c43fe501d831460ab89815f5e6228fcd5aaa84e7f5570"}
Sep 17 16:42:18.683830 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:18.683767    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" podStartSLOduration=3.281455762 podStartE2EDuration="5.683749638s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.402293876s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:18.681852745 +0000 UTC m=+3804.153692590" watchObservedRunningTime="2026-09-17 16:42:18.683749638 +0000 UTC m=+3804.155589483"
Sep 17 16:42:18.988868 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:18.988782    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:20.081918 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:20.081859    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:21.647063 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:21.647023    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:21.748080 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:21.748045    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:23.989050 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:23.989006    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:42:23.990921 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:23.990892    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 16:42:27.133855 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:27.133817    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:27.230771 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:27.230733    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:33.989946 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:33.989892    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 16:42:42.122550 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:42.122509    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:43.989535 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:43.989483    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 16:42:46.645947 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:46.645913    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"
Sep 17 16:42:46.709220 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:46.709179    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:46.885479 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:46.885418    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"]
Sep 17 16:42:46.889561 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:46.889540    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"
Sep 17 16:42:46.891989 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:46.891964    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 16:42:46.892115 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:46.891994    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-p5fqb\""
Sep 17 16:42:46.892115 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:46.892006    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 16:42:46.892253 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:46.892112    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-tpv2n\""
Sep 17 16:42:46.896493 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:46.892571    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 16:42:46.907814 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:46.907784    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"]
Sep 17 16:42:46.908325 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:46.908298    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:46.983195 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:46.983137    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fef696ab-429d-411b-aaf5-afa8d3c931ed-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc\" (UID: \"fef696ab-429d-411b-aaf5-afa8d3c931ed\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"
Sep 17 16:42:46.983365 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:46.983275    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5t6\" (UniqueName: \"kubernetes.io/projected/fef696ab-429d-411b-aaf5-afa8d3c931ed-kube-api-access-kd5t6\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc\" (UID: \"fef696ab-429d-411b-aaf5-afa8d3c931ed\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"
Sep 17 16:42:47.084090 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:47.084058    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fef696ab-429d-411b-aaf5-afa8d3c931ed-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc\" (UID: \"fef696ab-429d-411b-aaf5-afa8d3c931ed\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"
Sep 17 16:42:47.084279 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:47.084126    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5t6\" (UniqueName: \"kubernetes.io/projected/fef696ab-429d-411b-aaf5-afa8d3c931ed-kube-api-access-kd5t6\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc\" (UID: \"fef696ab-429d-411b-aaf5-afa8d3c931ed\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"
Sep 17 16:42:47.087138 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:47.087105    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fef696ab-429d-411b-aaf5-afa8d3c931ed-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc\" (UID: \"fef696ab-429d-411b-aaf5-afa8d3c931ed\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"
Sep 17 16:42:47.096570 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:47.096539    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5t6\" (UniqueName: \"kubernetes.io/projected/fef696ab-429d-411b-aaf5-afa8d3c931ed-kube-api-access-kd5t6\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc\" (UID: \"fef696ab-429d-411b-aaf5-afa8d3c931ed\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"
Sep 17 16:42:47.208288 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:47.208188    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"
Sep 17 16:42:47.497384 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:47.497353    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"]
Sep 17 16:42:47.499550 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:42:47.499521    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef696ab_429d_411b_aaf5_afa8d3c931ed.slice/crio-2cf638ec79840974b1d1fd31326da0d48843f10380bb275a8dfb8c37126f5476 WatchSource:0}: Error finding container 2cf638ec79840974b1d1fd31326da0d48843f10380bb275a8dfb8c37126f5476: Status 404 returned error can't find the container with id 2cf638ec79840974b1d1fd31326da0d48843f10380bb275a8dfb8c37126f5476
Sep 17 16:42:47.807794 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:47.807710    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc" event={"ID":"fef696ab-429d-411b-aaf5-afa8d3c931ed","Type":"ContainerStarted","Data":"582c516148cc1e56ade91e5aca9883c07f9db5cabbfc79b19846192d668923a6"}
Sep 17 16:42:47.807794 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:47.807768    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc" event={"ID":"fef696ab-429d-411b-aaf5-afa8d3c931ed","Type":"ContainerStarted","Data":"2cf638ec79840974b1d1fd31326da0d48843f10380bb275a8dfb8c37126f5476"}
Sep 17 16:42:47.808210 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:47.807842    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"
Sep 17 16:42:47.826994 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:47.826944    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc" podStartSLOduration=1.826927996 podStartE2EDuration="1.826927996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:47.825726614 +0000 UTC m=+3833.297566460" watchObservedRunningTime="2026-09-17 16:42:47.826927996 +0000 UTC m=+3833.298767842"
Sep 17 16:42:53.075401 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:53.075357    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:53.989459 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:53.989390    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 16:42:59.017422 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:59.017373    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:59.084580 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:42:59.084518    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:03.989759 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:03.989703    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 16:43:13.989350 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:13.989295    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 16:43:14.023705 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:14.023656    2574 scope.go:117] "RemoveContainer" containerID="e3530083d3fb637e3011f7b710821bd3b0ac8115ce1364b57fffa5dbe2835014"
Sep 17 16:43:18.816094 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:18.816014    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"
Sep 17 16:43:20.373083 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:20.373048    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:20.436150 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:20.436110    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"]
Sep 17 16:43:21.969665 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:21.969623    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:23.989428 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:23.989378    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 16:43:28.816456 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:28.816401    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc" podUID="fef696ab-429d-411b-aaf5-afa8d3c931ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:43:31.103550 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:31.103511    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:33.989433 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:33.989384    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 16:43:35.456335 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:35.456283    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc" podUID="fef696ab-429d-411b-aaf5-afa8d3c931ed" containerName="main" containerID="cri-o://582c516148cc1e56ade91e5aca9883c07f9db5cabbfc79b19846192d668923a6" gracePeriod=45
Sep 17 16:43:38.816504 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:38.816452    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc" podUID="fef696ab-429d-411b-aaf5-afa8d3c931ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:43:43.989599 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:43.989554    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 16:43:48.816214 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:48.816162    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc" podUID="fef696ab-429d-411b-aaf5-afa8d3c931ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:43:48.816725 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:48.816265    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"
Sep 17 16:43:53.989108 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:53.989052    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 16:43:58.816292 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:43:58.816249    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc" podUID="fef696ab-429d-411b-aaf5-afa8d3c931ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:44:03.989880 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:03.989821    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 16:44:05.705670 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:05.705640    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"
Sep 17 16:44:05.813936 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:05.813851    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fef696ab-429d-411b-aaf5-afa8d3c931ed-tls-certs\") pod \"fef696ab-429d-411b-aaf5-afa8d3c931ed\" (UID: \"fef696ab-429d-411b-aaf5-afa8d3c931ed\") "
Sep 17 16:44:05.813936 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:05.813915    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd5t6\" (UniqueName: \"kubernetes.io/projected/fef696ab-429d-411b-aaf5-afa8d3c931ed-kube-api-access-kd5t6\") pod \"fef696ab-429d-411b-aaf5-afa8d3c931ed\" (UID: \"fef696ab-429d-411b-aaf5-afa8d3c931ed\") "
Sep 17 16:44:05.816161 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:05.816116    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef696ab-429d-411b-aaf5-afa8d3c931ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fef696ab-429d-411b-aaf5-afa8d3c931ed" (UID: "fef696ab-429d-411b-aaf5-afa8d3c931ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:05.816161 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:05.816151    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef696ab-429d-411b-aaf5-afa8d3c931ed-kube-api-access-kd5t6" (OuterVolumeSpecName: "kube-api-access-kd5t6") pod "fef696ab-429d-411b-aaf5-afa8d3c931ed" (UID: "fef696ab-429d-411b-aaf5-afa8d3c931ed"). InnerVolumeSpecName "kube-api-access-kd5t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:44:05.915268 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:05.915231    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fef696ab-429d-411b-aaf5-afa8d3c931ed-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:44:05.915268 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:05.915262    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd5t6\" (UniqueName: \"kubernetes.io/projected/fef696ab-429d-411b-aaf5-afa8d3c931ed-kube-api-access-kd5t6\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:44:06.234139 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:06.234102    2574 generic.go:356] "Generic (PLEG): container finished" podID="fef696ab-429d-411b-aaf5-afa8d3c931ed" containerID="582c516148cc1e56ade91e5aca9883c07f9db5cabbfc79b19846192d668923a6" exitCode=0
Sep 17 16:44:06.234139 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:06.234148    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"
Sep 17 16:44:06.234412 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:06.234154    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc" event={"ID":"fef696ab-429d-411b-aaf5-afa8d3c931ed","Type":"ContainerDied","Data":"582c516148cc1e56ade91e5aca9883c07f9db5cabbfc79b19846192d668923a6"}
Sep 17 16:44:06.234412 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:06.234207    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc" event={"ID":"fef696ab-429d-411b-aaf5-afa8d3c931ed","Type":"ContainerDied","Data":"2cf638ec79840974b1d1fd31326da0d48843f10380bb275a8dfb8c37126f5476"}
Sep 17 16:44:06.234412 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:06.234228    2574 scope.go:117] "RemoveContainer" containerID="582c516148cc1e56ade91e5aca9883c07f9db5cabbfc79b19846192d668923a6"
Sep 17 16:44:06.244285 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:06.244265    2574 scope.go:117] "RemoveContainer" containerID="582c516148cc1e56ade91e5aca9883c07f9db5cabbfc79b19846192d668923a6"
Sep 17 16:44:06.244574 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:44:06.244553    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582c516148cc1e56ade91e5aca9883c07f9db5cabbfc79b19846192d668923a6\": container with ID starting with 582c516148cc1e56ade91e5aca9883c07f9db5cabbfc79b19846192d668923a6 not found: ID does not exist" containerID="582c516148cc1e56ade91e5aca9883c07f9db5cabbfc79b19846192d668923a6"
Sep 17 16:44:06.244688 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:06.244581    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582c516148cc1e56ade91e5aca9883c07f9db5cabbfc79b19846192d668923a6"} err="failed to get container status \"582c516148cc1e56ade91e5aca9883c07f9db5cabbfc79b19846192d668923a6\": rpc error: code = NotFound desc = could not find container \"582c516148cc1e56ade91e5aca9883c07f9db5cabbfc79b19846192d668923a6\": container with ID starting with 582c516148cc1e56ade91e5aca9883c07f9db5cabbfc79b19846192d668923a6 not found: ID does not exist"
Sep 17 16:44:06.259345 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:06.259307    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"]
Sep 17 16:44:06.261084 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:06.261058    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegq2cc"]
Sep 17 16:44:07.053181 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:07.053141    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fef696ab-429d-411b-aaf5-afa8d3c931ed" path="/var/lib/kubelet/pods/fef696ab-429d-411b-aaf5-afa8d3c931ed/volumes"
Sep 17 16:44:10.403171 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:10.403134    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:10.497405 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:10.497363    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:13.998940 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:13.998899    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:44:14.006674 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:14.006640    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:44:16.923562 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:16.923520    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:18.987578 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:18.987544    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"]
Sep 17 16:44:23.168263 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:23.168221    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:23.242501 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:23.242464    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:26.048528 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:26.048487    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:26.251428 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:26.251388    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:34.010215 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.010167    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main" containerID="cri-o://f573471fee28c793ad5c43fe501d831460ab89815f5e6228fcd5aaa84e7f5570" gracePeriod=45
Sep 17 16:44:34.758587 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.758557    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:44:34.869182 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.869150    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-cabundle-cert\") pod \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") "
Sep 17 16:44:34.869390 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.869197    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-tmp-dir\") pod \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") "
Sep 17 16:44:34.869390 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.869220    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-kserve-provision-location\") pod \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") "
Sep 17 16:44:34.869390 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.869250    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-tls-certs\") pod \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") "
Sep 17 16:44:34.869390 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.869269    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-home\") pod \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") "
Sep 17 16:44:34.869390 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.869315    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95zwh\" (UniqueName: \"kubernetes.io/projected/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-kube-api-access-95zwh\") pod \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") "
Sep 17 16:44:34.869700 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.869398    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-dshm\") pod \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") "
Sep 17 16:44:34.869700 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.869488    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-model-cache\") pod \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\" (UID: \"9c17bbec-7fea-44e3-9202-e2ca21ad9d33\") "
Sep 17 16:44:34.869700 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.869642    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9c17bbec-7fea-44e3-9202-e2ca21ad9d33" (UID: "9c17bbec-7fea-44e3-9202-e2ca21ad9d33"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:34.869866 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.869845    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c17bbec-7fea-44e3-9202-e2ca21ad9d33" (UID: "9c17bbec-7fea-44e3-9202-e2ca21ad9d33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:44:34.869938 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.869915    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-cabundle-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:44:34.870082 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.870055    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-home" (OuterVolumeSpecName: "home") pod "9c17bbec-7fea-44e3-9202-e2ca21ad9d33" (UID: "9c17bbec-7fea-44e3-9202-e2ca21ad9d33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:44:34.872409 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.872373    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c17bbec-7fea-44e3-9202-e2ca21ad9d33" (UID: "9c17bbec-7fea-44e3-9202-e2ca21ad9d33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:34.872608 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.872571    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-kube-api-access-95zwh" (OuterVolumeSpecName: "kube-api-access-95zwh") pod "9c17bbec-7fea-44e3-9202-e2ca21ad9d33" (UID: "9c17bbec-7fea-44e3-9202-e2ca21ad9d33"). InnerVolumeSpecName "kube-api-access-95zwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:44:34.872860 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.872840    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-dshm" (OuterVolumeSpecName: "dshm") pod "9c17bbec-7fea-44e3-9202-e2ca21ad9d33" (UID: "9c17bbec-7fea-44e3-9202-e2ca21ad9d33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:44:34.882188 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.882159    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c17bbec-7fea-44e3-9202-e2ca21ad9d33" (UID: "9c17bbec-7fea-44e3-9202-e2ca21ad9d33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:44:34.894037 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.894010    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c17bbec-7fea-44e3-9202-e2ca21ad9d33" (UID: "9c17bbec-7fea-44e3-9202-e2ca21ad9d33"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:44:34.970853 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.970812    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:44:34.970853 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.970845    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-kserve-provision-location\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:44:34.970853 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.970856    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:44:34.970853 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.970866    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:44:34.971134 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.970875    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95zwh\" (UniqueName: \"kubernetes.io/projected/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-kube-api-access-95zwh\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:44:34.971134 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.970883    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:44:34.971134 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:34.970891    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c17bbec-7fea-44e3-9202-e2ca21ad9d33-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.394185 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:35.394150    2574 generic.go:356] "Generic (PLEG): container finished" podID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerID="f573471fee28c793ad5c43fe501d831460ab89815f5e6228fcd5aaa84e7f5570" exitCode=0
Sep 17 16:44:35.394658 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:35.394197    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" event={"ID":"9c17bbec-7fea-44e3-9202-e2ca21ad9d33","Type":"ContainerDied","Data":"f573471fee28c793ad5c43fe501d831460ab89815f5e6228fcd5aaa84e7f5570"}
Sep 17 16:44:35.394658 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:35.394203    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"
Sep 17 16:44:35.394658 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:35.394238    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx" event={"ID":"9c17bbec-7fea-44e3-9202-e2ca21ad9d33","Type":"ContainerDied","Data":"f6b24b8e49209caff9e198789bdc307dedd4aeed37c06db950a81d6cd1f452ea"}
Sep 17 16:44:35.394658 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:35.394260    2574 scope.go:117] "RemoveContainer" containerID="f573471fee28c793ad5c43fe501d831460ab89815f5e6228fcd5aaa84e7f5570"
Sep 17 16:44:35.403124 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:35.403097    2574 scope.go:117] "RemoveContainer" containerID="18340be96d6759c28717ba34fd4b5deaabd2994f150e44eec0229cd980c419b6"
Sep 17 16:44:35.411122 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:35.411103    2574 scope.go:117] "RemoveContainer" containerID="f573471fee28c793ad5c43fe501d831460ab89815f5e6228fcd5aaa84e7f5570"
Sep 17 16:44:35.411424 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:44:35.411403    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f573471fee28c793ad5c43fe501d831460ab89815f5e6228fcd5aaa84e7f5570\": container with ID starting with f573471fee28c793ad5c43fe501d831460ab89815f5e6228fcd5aaa84e7f5570 not found: ID does not exist" containerID="f573471fee28c793ad5c43fe501d831460ab89815f5e6228fcd5aaa84e7f5570"
Sep 17 16:44:35.411506 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:35.411434    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f573471fee28c793ad5c43fe501d831460ab89815f5e6228fcd5aaa84e7f5570"} err="failed to get container status \"f573471fee28c793ad5c43fe501d831460ab89815f5e6228fcd5aaa84e7f5570\": rpc error: code = NotFound desc = could not find container \"f573471fee28c793ad5c43fe501d831460ab89815f5e6228fcd5aaa84e7f5570\": container with ID starting with f573471fee28c793ad5c43fe501d831460ab89815f5e6228fcd5aaa84e7f5570 not found: ID does not exist"
Sep 17 16:44:35.411506 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:35.411469    2574 scope.go:117] "RemoveContainer" containerID="18340be96d6759c28717ba34fd4b5deaabd2994f150e44eec0229cd980c419b6"
Sep 17 16:44:35.411707 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:44:35.411690    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18340be96d6759c28717ba34fd4b5deaabd2994f150e44eec0229cd980c419b6\": container with ID starting with 18340be96d6759c28717ba34fd4b5deaabd2994f150e44eec0229cd980c419b6 not found: ID does not exist" containerID="18340be96d6759c28717ba34fd4b5deaabd2994f150e44eec0229cd980c419b6"
Sep 17 16:44:35.411757 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:35.411712    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18340be96d6759c28717ba34fd4b5deaabd2994f150e44eec0229cd980c419b6"} err="failed to get container status \"18340be96d6759c28717ba34fd4b5deaabd2994f150e44eec0229cd980c419b6\": rpc error: code = NotFound desc = could not find container \"18340be96d6759c28717ba34fd4b5deaabd2994f150e44eec0229cd980c419b6\": container with ID starting with 18340be96d6759c28717ba34fd4b5deaabd2994f150e44eec0229cd980c419b6 not found: ID does not exist"
Sep 17 16:44:35.428041 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:35.428003    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"]
Sep 17 16:44:35.439206 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:35.439171    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6989b4f56b-c9xkx"]
Sep 17 16:44:37.049408 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:37.049372    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" path="/var/lib/kubelet/pods/9c17bbec-7fea-44e3-9202-e2ca21ad9d33/volumes"
Sep 17 16:44:39.818844 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:39.818808    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:41.196212 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:41.196170    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"]
Sep 17 16:44:41.200606 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:41.200408    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:41.271953 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:41.271922    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:46.645841 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:46.645797    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66" podUID="2218528e-4c40-4e46-b97f-52f102c1a1e0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:44:56.218911 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:56.218872    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66" podUID="2218528e-4c40-4e46-b97f-52f102c1a1e0" containerName="main" containerID="cri-o://392ea308d3733bbde78ef8cec36dee33342a52bd0e1eb59fb5e334713f0e4b5e" gracePeriod=45
Sep 17 16:44:56.645661 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:44:56.645621    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66" podUID="2218528e-4c40-4e46-b97f-52f102c1a1e0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:45:00.328068 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:00.328028    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:00.427006 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:00.426969    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:06.645378 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:06.645340    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66" podUID="2218528e-4c40-4e46-b97f-52f102c1a1e0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:45:06.645805 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:06.645479    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"
Sep 17 16:45:11.921557 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:11.921509    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:12.021921 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:12.021884    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:16.646081 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:16.646040    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66" podUID="2218528e-4c40-4e46-b97f-52f102c1a1e0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:45:26.470203 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.470171    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"
Sep 17 16:45:26.498899 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.498806    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2218528e-4c40-4e46-b97f-52f102c1a1e0-tls-certs\") pod \"2218528e-4c40-4e46-b97f-52f102c1a1e0\" (UID: \"2218528e-4c40-4e46-b97f-52f102c1a1e0\") "
Sep 17 16:45:26.501032 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.501005    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2218528e-4c40-4e46-b97f-52f102c1a1e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2218528e-4c40-4e46-b97f-52f102c1a1e0" (UID: "2218528e-4c40-4e46-b97f-52f102c1a1e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:45:26.599398 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.599364    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbtzh\" (UniqueName: \"kubernetes.io/projected/2218528e-4c40-4e46-b97f-52f102c1a1e0-kube-api-access-fbtzh\") pod \"2218528e-4c40-4e46-b97f-52f102c1a1e0\" (UID: \"2218528e-4c40-4e46-b97f-52f102c1a1e0\") "
Sep 17 16:45:26.599578 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.599559    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2218528e-4c40-4e46-b97f-52f102c1a1e0-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:45:26.601396 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.601372    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2218528e-4c40-4e46-b97f-52f102c1a1e0-kube-api-access-fbtzh" (OuterVolumeSpecName: "kube-api-access-fbtzh") pod "2218528e-4c40-4e46-b97f-52f102c1a1e0" (UID: "2218528e-4c40-4e46-b97f-52f102c1a1e0"). InnerVolumeSpecName "kube-api-access-fbtzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:45:26.634623 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.634591    2574 generic.go:356] "Generic (PLEG): container finished" podID="2218528e-4c40-4e46-b97f-52f102c1a1e0" containerID="392ea308d3733bbde78ef8cec36dee33342a52bd0e1eb59fb5e334713f0e4b5e" exitCode=0
Sep 17 16:45:26.634872 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.634631    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66" event={"ID":"2218528e-4c40-4e46-b97f-52f102c1a1e0","Type":"ContainerDied","Data":"392ea308d3733bbde78ef8cec36dee33342a52bd0e1eb59fb5e334713f0e4b5e"}
Sep 17 16:45:26.634872 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.634642    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"
Sep 17 16:45:26.634872 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.634674    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66" event={"ID":"2218528e-4c40-4e46-b97f-52f102c1a1e0","Type":"ContainerDied","Data":"1721926d5e983495201f12005b91da3157977ec0b41956687a170b5b270bd107"}
Sep 17 16:45:26.634872 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.634696    2574 scope.go:117] "RemoveContainer" containerID="392ea308d3733bbde78ef8cec36dee33342a52bd0e1eb59fb5e334713f0e4b5e"
Sep 17 16:45:26.644639 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.644620    2574 scope.go:117] "RemoveContainer" containerID="392ea308d3733bbde78ef8cec36dee33342a52bd0e1eb59fb5e334713f0e4b5e"
Sep 17 16:45:26.644913 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:45:26.644894    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392ea308d3733bbde78ef8cec36dee33342a52bd0e1eb59fb5e334713f0e4b5e\": container with ID starting with 392ea308d3733bbde78ef8cec36dee33342a52bd0e1eb59fb5e334713f0e4b5e not found: ID does not exist" containerID="392ea308d3733bbde78ef8cec36dee33342a52bd0e1eb59fb5e334713f0e4b5e"
Sep 17 16:45:26.644982 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.644920    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392ea308d3733bbde78ef8cec36dee33342a52bd0e1eb59fb5e334713f0e4b5e"} err="failed to get container status \"392ea308d3733bbde78ef8cec36dee33342a52bd0e1eb59fb5e334713f0e4b5e\": rpc error: code = NotFound desc = could not find container \"392ea308d3733bbde78ef8cec36dee33342a52bd0e1eb59fb5e334713f0e4b5e\": container with ID starting with 392ea308d3733bbde78ef8cec36dee33342a52bd0e1eb59fb5e334713f0e4b5e not found: ID does not exist"
Sep 17 16:45:26.661882 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.661854    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"]
Sep 17 16:45:26.665356 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.665326    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-f7x66"]
Sep 17 16:45:26.700083 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:26.700059    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbtzh\" (UniqueName: \"kubernetes.io/projected/2218528e-4c40-4e46-b97f-52f102c1a1e0-kube-api-access-fbtzh\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:45:27.048721 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:27.048682    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2218528e-4c40-4e46-b97f-52f102c1a1e0" path="/var/lib/kubelet/pods/2218528e-4c40-4e46-b97f-52f102c1a1e0/volumes"
Sep 17 16:45:47.723716 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:47.723677    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:47.855880 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:47.855842    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:49.485302 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:49.485256    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:53.217854 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.217813    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"]
Sep 17 16:45:53.218234 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.218208    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2218528e-4c40-4e46-b97f-52f102c1a1e0" containerName="main"
Sep 17 16:45:53.218234 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.218220    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2218528e-4c40-4e46-b97f-52f102c1a1e0" containerName="main"
Sep 17 16:45:53.218308 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.218237    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fef696ab-429d-411b-aaf5-afa8d3c931ed" containerName="main"
Sep 17 16:45:53.218308 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.218243    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fef696ab-429d-411b-aaf5-afa8d3c931ed" containerName="main"
Sep 17 16:45:53.218308 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.218252    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="storage-initializer"
Sep 17 16:45:53.218308 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.218258    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="storage-initializer"
Sep 17 16:45:53.218308 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.218268    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main"
Sep 17 16:45:53.218308 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.218273    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main"
Sep 17 16:45:53.218516 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.218327    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c17bbec-7fea-44e3-9202-e2ca21ad9d33" containerName="main"
Sep 17 16:45:53.218516 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.218336    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2218528e-4c40-4e46-b97f-52f102c1a1e0" containerName="main"
Sep 17 16:45:53.218516 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.218345    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fef696ab-429d-411b-aaf5-afa8d3c931ed" containerName="main"
Sep 17 16:45:53.221341 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.221313    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.224673 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.224652    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 16:45:53.224782 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.224689    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-6qjxj\""
Sep 17 16:45:53.224782 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.224655    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 16:45:53.224910 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.224654    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 16:45:53.234923 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.234901    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"]
Sep 17 16:45:53.259674 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.259637    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:53.295846 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.295817    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.295944 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.295861    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.295944 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.295903    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqjjm\" (UniqueName: \"kubernetes.io/projected/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-kube-api-access-tqjjm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.296025 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.295963    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.296025 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.296001    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.296025 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.296019    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.396500 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.396461    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.396500 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.396507    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.396746 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.396526    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.396746 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.396603    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.396746 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.396651    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.396746 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.396685    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqjjm\" (UniqueName: \"kubernetes.io/projected/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-kube-api-access-tqjjm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.396967 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.396940    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.397010 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.396971    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.397050 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.397020    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.398983 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.398953    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.399375 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.399355    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.406026 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.405997    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqjjm\" (UniqueName: \"kubernetes.io/projected/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-kube-api-access-tqjjm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-58wss\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.503601 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.503520    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"]
Sep 17 16:45:53.507204 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.507176    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"
Sep 17 16:45:53.511166 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.511138    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-g5wtm\""
Sep 17 16:45:53.526343 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.526316    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"]
Sep 17 16:45:53.531330 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.531306    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:53.548843 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.548809    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:53.598890 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.598685    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff61e897-068e-4b0d-9ba2-74b2cd33a081-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx\" (UID: \"ff61e897-068e-4b0d-9ba2-74b2cd33a081\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"
Sep 17 16:45:53.598890 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.598813    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6b8v\" (UniqueName: \"kubernetes.io/projected/ff61e897-068e-4b0d-9ba2-74b2cd33a081-kube-api-access-d6b8v\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx\" (UID: \"ff61e897-068e-4b0d-9ba2-74b2cd33a081\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"
Sep 17 16:45:53.700164 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.700129    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff61e897-068e-4b0d-9ba2-74b2cd33a081-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx\" (UID: \"ff61e897-068e-4b0d-9ba2-74b2cd33a081\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"
Sep 17 16:45:53.700367 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.700205    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6b8v\" (UniqueName: \"kubernetes.io/projected/ff61e897-068e-4b0d-9ba2-74b2cd33a081-kube-api-access-d6b8v\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx\" (UID: \"ff61e897-068e-4b0d-9ba2-74b2cd33a081\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"
Sep 17 16:45:53.702954 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.702918    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff61e897-068e-4b0d-9ba2-74b2cd33a081-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx\" (UID: \"ff61e897-068e-4b0d-9ba2-74b2cd33a081\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"
Sep 17 16:45:53.710545 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.710519    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6b8v\" (UniqueName: \"kubernetes.io/projected/ff61e897-068e-4b0d-9ba2-74b2cd33a081-kube-api-access-d6b8v\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx\" (UID: \"ff61e897-068e-4b0d-9ba2-74b2cd33a081\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"
Sep 17 16:45:53.815413 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.815385    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"]
Sep 17 16:45:53.817979 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:45:53.817941    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c41b4ab_93b6_4dfc_ac94_02b92f4e9efa.slice/crio-5021fb39fbd204bdd06108c88bd9a986bbd0e33316c3ac3bf04f1a1a663f10b4 WatchSource:0}: Error finding container 5021fb39fbd204bdd06108c88bd9a986bbd0e33316c3ac3bf04f1a1a663f10b4: Status 404 returned error can't find the container with id 5021fb39fbd204bdd06108c88bd9a986bbd0e33316c3ac3bf04f1a1a663f10b4
Sep 17 16:45:53.821734 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:53.821711    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"
Sep 17 16:45:54.116016 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:54.115987    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"]
Sep 17 16:45:54.116983 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:45:54.116957    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff61e897_068e_4b0d_9ba2_74b2cd33a081.slice/crio-1c1fefce26bfd5667d9a57fc813b5a4ee62cd7a909779bec45fa00b3a2089dc6 WatchSource:0}: Error finding container 1c1fefce26bfd5667d9a57fc813b5a4ee62cd7a909779bec45fa00b3a2089dc6: Status 404 returned error can't find the container with id 1c1fefce26bfd5667d9a57fc813b5a4ee62cd7a909779bec45fa00b3a2089dc6
Sep 17 16:45:54.768525 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:54.768487    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx" event={"ID":"ff61e897-068e-4b0d-9ba2-74b2cd33a081","Type":"ContainerStarted","Data":"3b339e1b0a33a4a8a808b66bde48f68b9399b25df29ff5808fa15fdb55a9e2f6"}
Sep 17 16:45:54.769004 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:54.768639    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"
Sep 17 16:45:54.769004 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:54.768686    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx" event={"ID":"ff61e897-068e-4b0d-9ba2-74b2cd33a081","Type":"ContainerStarted","Data":"1c1fefce26bfd5667d9a57fc813b5a4ee62cd7a909779bec45fa00b3a2089dc6"}
Sep 17 16:45:54.770060 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:54.770035    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss" event={"ID":"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa","Type":"ContainerStarted","Data":"4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a"}
Sep 17 16:45:54.770182 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:54.770079    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss" event={"ID":"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa","Type":"ContainerStarted","Data":"5021fb39fbd204bdd06108c88bd9a986bbd0e33316c3ac3bf04f1a1a663f10b4"}
Sep 17 16:45:54.789801 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:54.789758    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx" podStartSLOduration=1.7897451960000001 podStartE2EDuration="1.789745196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:45:54.787640667 +0000 UTC m=+4020.259480515" watchObservedRunningTime="2026-09-17 16:45:54.789745196 +0000 UTC m=+4020.261585042"
Sep 17 16:45:54.805887 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:54.805834    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss" podStartSLOduration=1.805815314 podStartE2EDuration="1.805815314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:45:54.805418535 +0000 UTC m=+4020.277258394" watchObservedRunningTime="2026-09-17 16:45:54.805815314 +0000 UTC m=+4020.277655164"
Sep 17 16:45:55.531844 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:55.531809    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:45:55.849009 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:55.848924    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:56.930145 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:45:56.930104    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:02.393525 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:02.393487    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:02.576288 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:02.576248    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:03.531451 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:03.531406    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:46:03.543883 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:03.543855    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:46:03.584433 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:03.584398    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:03.656988 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:03.656955    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:03.826983 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:03.826905    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:46:16.206627 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:16.206581    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:25.777691 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:25.777614    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"
Sep 17 16:46:38.950317 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:38.950279    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"]
Sep 17 16:46:38.959862 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:46:38.959820    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 16:46:38.959862 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:46:38.959862 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:46:38.959862 ip-10-0-129-37 kubenswrapper[2574]:  > containerID="4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a" cmd=["/bin/sleep","15"]
Sep 17 16:46:38.960302 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:38.959882    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss" message="" err=<
Sep 17 16:46:38.960302 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:46:38.960302 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:46:38.960302 ip-10-0-129-37 kubenswrapper[2574]:  >
Sep 17 16:46:38.960302 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:46:38.959898    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 16:46:38.960302 ip-10-0-129-37 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 16:46:38.960302 ip-10-0-129-37 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 16:46:38.960302 ip-10-0-129-37 kubenswrapper[2574]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss" podUID="8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa" containerName="main" containerID="cri-o://4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a"
Sep 17 16:46:38.960302 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:38.959962    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss" podUID="8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa" containerName="main" containerID="cri-o://4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a" gracePeriod=60
Sep 17 16:46:39.707007 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.706978    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:46:39.803739 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.803705    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-tls-certs\") pod \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") "
Sep 17 16:46:39.803739 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.803743    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-dshm\") pod \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") "
Sep 17 16:46:39.804005 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.803764    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-home\") pod \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") "
Sep 17 16:46:39.804005 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.803786    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-tmp-dir\") pod \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") "
Sep 17 16:46:39.804005 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.803837    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-model-cache\") pod \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") "
Sep 17 16:46:39.804005 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.803891    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqjjm\" (UniqueName: \"kubernetes.io/projected/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-kube-api-access-tqjjm\") pod \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\" (UID: \"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa\") "
Sep 17 16:46:39.804195 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.804096    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-home" (OuterVolumeSpecName: "home") pod "8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa" (UID: "8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:46:39.804195 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.804111    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa" (UID: "8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:46:39.804195 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.804142    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa" (UID: "8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:46:39.806157 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.806133    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-kube-api-access-tqjjm" (OuterVolumeSpecName: "kube-api-access-tqjjm") pod "8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa" (UID: "8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa"). InnerVolumeSpecName "kube-api-access-tqjjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:46:39.806250 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.806156    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa" (UID: "8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:46:39.806250 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.806217    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-dshm" (OuterVolumeSpecName: "dshm") pod "8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa" (UID: "8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:46:39.904891 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.904855    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:46:39.904891 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.904885    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqjjm\" (UniqueName: \"kubernetes.io/projected/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-kube-api-access-tqjjm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:46:39.904891 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.904895    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:46:39.905133 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.904906    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:46:39.905133 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.904913    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:46:39.905133 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.904940    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:46:39.990720 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.990681    2574 generic.go:356] "Generic (PLEG): container finished" podID="8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa" containerID="4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a" exitCode=0
Sep 17 16:46:39.991182 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.990726    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"
Sep 17 16:46:39.991182 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.990738    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss" event={"ID":"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa","Type":"ContainerDied","Data":"4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a"}
Sep 17 16:46:39.991182 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.990793    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss" event={"ID":"8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa","Type":"ContainerDied","Data":"5021fb39fbd204bdd06108c88bd9a986bbd0e33316c3ac3bf04f1a1a663f10b4"}
Sep 17 16:46:39.991182 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:39.990823    2574 scope.go:117] "RemoveContainer" containerID="4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a"
Sep 17 16:46:40.000645 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:40.000623    2574 scope.go:117] "RemoveContainer" containerID="4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a"
Sep 17 16:46:40.000892 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:46:40.000875    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a\": container with ID starting with 4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a not found: ID does not exist" containerID="4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a"
Sep 17 16:46:40.000938 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:40.000900    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a"} err="failed to get container status \"4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a\": rpc error: code = NotFound desc = could not find container \"4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a\": container with ID starting with 4b957a63c793d163ae4a201e8b9f69c880477c65047a32bcd25d6a629b26f87a not found: ID does not exist"
Sep 17 16:46:40.012468 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:40.012425    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"]
Sep 17 16:46:40.016408 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:40.016376    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-58wss"]
Sep 17 16:46:41.049127 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:41.049087    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa" path="/var/lib/kubelet/pods/8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa/volumes"
Sep 17 16:46:42.794073 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:42.794035    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"]
Sep 17 16:46:42.813423 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:42.813363    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:42.913524 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:42.913459    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:45.776809 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:45.776772    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx" podUID="ff61e897-068e-4b0d-9ba2-74b2cd33a081" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:46:55.777140 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:55.777099    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx" podUID="ff61e897-068e-4b0d-9ba2-74b2cd33a081" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:46:57.815928 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:46:57.815865    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx" podUID="ff61e897-068e-4b0d-9ba2-74b2cd33a081" containerName="main" containerID="cri-o://3b339e1b0a33a4a8a808b66bde48f68b9399b25df29ff5808fa15fdb55a9e2f6" gracePeriod=45
Sep 17 16:47:02.357245 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:02.357204    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:05.777303 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:05.777261    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx" podUID="ff61e897-068e-4b0d-9ba2-74b2cd33a081" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:47:05.777724 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:05.777374    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"
Sep 17 16:47:15.777675 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:15.777631    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx" podUID="ff61e897-068e-4b0d-9ba2-74b2cd33a081" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:47:25.777545 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:25.777503    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx" podUID="ff61e897-068e-4b0d-9ba2-74b2cd33a081" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 16:47:28.081469 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.081389    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"
Sep 17 16:47:28.139827 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.139789    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff61e897-068e-4b0d-9ba2-74b2cd33a081-tls-certs\") pod \"ff61e897-068e-4b0d-9ba2-74b2cd33a081\" (UID: \"ff61e897-068e-4b0d-9ba2-74b2cd33a081\") "
Sep 17 16:47:28.140016 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.139896    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6b8v\" (UniqueName: \"kubernetes.io/projected/ff61e897-068e-4b0d-9ba2-74b2cd33a081-kube-api-access-d6b8v\") pod \"ff61e897-068e-4b0d-9ba2-74b2cd33a081\" (UID: \"ff61e897-068e-4b0d-9ba2-74b2cd33a081\") "
Sep 17 16:47:28.142033 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.142007    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff61e897-068e-4b0d-9ba2-74b2cd33a081-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff61e897-068e-4b0d-9ba2-74b2cd33a081" (UID: "ff61e897-068e-4b0d-9ba2-74b2cd33a081"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:47:28.142136 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.142049    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff61e897-068e-4b0d-9ba2-74b2cd33a081-kube-api-access-d6b8v" (OuterVolumeSpecName: "kube-api-access-d6b8v") pod "ff61e897-068e-4b0d-9ba2-74b2cd33a081" (UID: "ff61e897-068e-4b0d-9ba2-74b2cd33a081"). InnerVolumeSpecName "kube-api-access-d6b8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:47:28.218666 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.218626    2574 generic.go:356] "Generic (PLEG): container finished" podID="ff61e897-068e-4b0d-9ba2-74b2cd33a081" containerID="3b339e1b0a33a4a8a808b66bde48f68b9399b25df29ff5808fa15fdb55a9e2f6" exitCode=0
Sep 17 16:47:28.218832 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.218673    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"
Sep 17 16:47:28.218832 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.218680    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx" event={"ID":"ff61e897-068e-4b0d-9ba2-74b2cd33a081","Type":"ContainerDied","Data":"3b339e1b0a33a4a8a808b66bde48f68b9399b25df29ff5808fa15fdb55a9e2f6"}
Sep 17 16:47:28.218832 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.218802    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx" event={"ID":"ff61e897-068e-4b0d-9ba2-74b2cd33a081","Type":"ContainerDied","Data":"1c1fefce26bfd5667d9a57fc813b5a4ee62cd7a909779bec45fa00b3a2089dc6"}
Sep 17 16:47:28.218832 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.218831    2574 scope.go:117] "RemoveContainer" containerID="3b339e1b0a33a4a8a808b66bde48f68b9399b25df29ff5808fa15fdb55a9e2f6"
Sep 17 16:47:28.227639 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.227621    2574 scope.go:117] "RemoveContainer" containerID="3b339e1b0a33a4a8a808b66bde48f68b9399b25df29ff5808fa15fdb55a9e2f6"
Sep 17 16:47:28.227904 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:47:28.227883    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b339e1b0a33a4a8a808b66bde48f68b9399b25df29ff5808fa15fdb55a9e2f6\": container with ID starting with 3b339e1b0a33a4a8a808b66bde48f68b9399b25df29ff5808fa15fdb55a9e2f6 not found: ID does not exist" containerID="3b339e1b0a33a4a8a808b66bde48f68b9399b25df29ff5808fa15fdb55a9e2f6"
Sep 17 16:47:28.227944 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.227913    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b339e1b0a33a4a8a808b66bde48f68b9399b25df29ff5808fa15fdb55a9e2f6"} err="failed to get container status \"3b339e1b0a33a4a8a808b66bde48f68b9399b25df29ff5808fa15fdb55a9e2f6\": rpc error: code = NotFound desc = could not find container \"3b339e1b0a33a4a8a808b66bde48f68b9399b25df29ff5808fa15fdb55a9e2f6\": container with ID starting with 3b339e1b0a33a4a8a808b66bde48f68b9399b25df29ff5808fa15fdb55a9e2f6 not found: ID does not exist"
Sep 17 16:47:28.240612 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.240590    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6b8v\" (UniqueName: \"kubernetes.io/projected/ff61e897-068e-4b0d-9ba2-74b2cd33a081-kube-api-access-d6b8v\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:47:28.240706 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.240613    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff61e897-068e-4b0d-9ba2-74b2cd33a081-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:47:28.244023 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.244001    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"]
Sep 17 16:47:28.250057 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:28.250029    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-mhqlx"]
Sep 17 16:47:29.049820 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:29.049785    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff61e897-068e-4b0d-9ba2-74b2cd33a081" path="/var/lib/kubelet/pods/ff61e897-068e-4b0d-9ba2-74b2cd33a081/volumes"
Sep 17 16:47:30.085320 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:30.085278    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:31.006288 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.006248    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7v7rc/must-gather-8qb9d"]
Sep 17 16:47:31.006664 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.006649    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff61e897-068e-4b0d-9ba2-74b2cd33a081" containerName="main"
Sep 17 16:47:31.006724 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.006666    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff61e897-068e-4b0d-9ba2-74b2cd33a081" containerName="main"
Sep 17 16:47:31.006724 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.006683    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa" containerName="main"
Sep 17 16:47:31.006724 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.006688    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa" containerName="main"
Sep 17 16:47:31.006824 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.006750    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c41b4ab-93b6-4dfc-ac94-02b92f4e9efa" containerName="main"
Sep 17 16:47:31.006824 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.006764    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff61e897-068e-4b0d-9ba2-74b2cd33a081" containerName="main"
Sep 17 16:47:31.011234 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.011213    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v7rc/must-gather-8qb9d"
Sep 17 16:47:31.013633 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.013607    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7v7rc\"/\"default-dockercfg-l5j9f\""
Sep 17 16:47:31.013773 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.013646    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7v7rc\"/\"openshift-service-ca.crt\""
Sep 17 16:47:31.013773 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.013712    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7v7rc\"/\"kube-root-ca.crt\""
Sep 17 16:47:31.020984 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.020949    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v7rc/must-gather-8qb9d"]
Sep 17 16:47:31.023759 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.023729    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:31.063710 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.063674    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb-must-gather-output\") pod \"must-gather-8qb9d\" (UID: \"9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb\") " pod="openshift-must-gather-7v7rc/must-gather-8qb9d"
Sep 17 16:47:31.063876 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.063735    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t758\" (UniqueName: \"kubernetes.io/projected/9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb-kube-api-access-6t758\") pod \"must-gather-8qb9d\" (UID: \"9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb\") " pod="openshift-must-gather-7v7rc/must-gather-8qb9d"
Sep 17 16:47:31.122397 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.122356    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:31.164880 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.164842    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb-must-gather-output\") pod \"must-gather-8qb9d\" (UID: \"9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb\") " pod="openshift-must-gather-7v7rc/must-gather-8qb9d"
Sep 17 16:47:31.165065 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.164914    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t758\" (UniqueName: \"kubernetes.io/projected/9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb-kube-api-access-6t758\") pod \"must-gather-8qb9d\" (UID: \"9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb\") " pod="openshift-must-gather-7v7rc/must-gather-8qb9d"
Sep 17 16:47:31.165268 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.165242    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb-must-gather-output\") pod \"must-gather-8qb9d\" (UID: \"9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb\") " pod="openshift-must-gather-7v7rc/must-gather-8qb9d"
Sep 17 16:47:31.173352 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.173325    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t758\" (UniqueName: \"kubernetes.io/projected/9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb-kube-api-access-6t758\") pod \"must-gather-8qb9d\" (UID: \"9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb\") " pod="openshift-must-gather-7v7rc/must-gather-8qb9d"
Sep 17 16:47:31.321584 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.321497    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v7rc/must-gather-8qb9d"
Sep 17 16:47:31.613859 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.613820    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v7rc/must-gather-8qb9d"]
Sep 17 16:47:31.616833 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:47:31.616805    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a1a8772_e8f4_4bd9_9bdb_c9a1f4504dbb.slice/crio-1e9f143e244a99ea7016d232ecb1297d27a6239dfdc6e5b09f5f15cfeef0860d WatchSource:0}: Error finding container 1e9f143e244a99ea7016d232ecb1297d27a6239dfdc6e5b09f5f15cfeef0860d: Status 404 returned error can't find the container with id 1e9f143e244a99ea7016d232ecb1297d27a6239dfdc6e5b09f5f15cfeef0860d
Sep 17 16:47:31.619187 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.619169    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:47:31.912816 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:31.912714    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:32.241283 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:32.241190    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v7rc/must-gather-8qb9d" event={"ID":"9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb","Type":"ContainerStarted","Data":"1e9f143e244a99ea7016d232ecb1297d27a6239dfdc6e5b09f5f15cfeef0860d"}
Sep 17 16:47:37.000825 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:37.000787    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:38.278744 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:38.278696    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v7rc/must-gather-8qb9d" event={"ID":"9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb","Type":"ContainerStarted","Data":"691d3c304c773acbc646c25e4005a707fbe6603e2f693a1cf57ebd55fe20f3fa"}
Sep 17 16:47:38.279228 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:38.278760    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v7rc/must-gather-8qb9d" event={"ID":"9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb","Type":"ContainerStarted","Data":"5f7ae203531df58e871f72742ddf66bd6dc725c8937057fb86b814df0d5fe9b1"}
Sep 17 16:47:38.303096 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:38.298318    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7v7rc/must-gather-8qb9d" podStartSLOduration=1.9392678810000001 podStartE2EDuration="8.298295408s" totalImagesPullingTime="6.359027527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:47:38.294951934 +0000 UTC m=+4123.766791784" watchObservedRunningTime="2026-09-17 16:47:38.298295408 +0000 UTC m=+4123.770135258"
Sep 17 16:47:44.725463 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:44.725409    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:44.827392 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:44.827347    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:56.865125 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:56.865046    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64b88459dd-xdnj5_1d1c6522-62ea-4c17-b992-33f236073899/router/0.log"
Sep 17 16:47:57.826867 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:57.826835    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64b88459dd-xdnj5_1d1c6522-62ea-4c17-b992-33f236073899/router/0.log"
Sep 17 16:47:59.046514 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:59.046479    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-26g8k_d06c1525-0f0c-4b12-b6ef-9b2a297400af/manager/0.log"
Sep 17 16:47:59.245103 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:59.245056    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-szvjq_ca00977e-3c20-4628-ae3d-125aa5d1756e/manager/0.log"
Sep 17 16:47:59.257609 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:59.257578    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-892hx_1facedd8-3eae-45ce-9516-21618b921ea5/limitador/0.log"
Sep 17 16:47:59.275555 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:47:59.275520    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qxkqk_e5f51b66-517d-4073-82ad-95511a0cca5d/manager/0.log"
Sep 17 16:48:00.401389 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:00.401351    2574 generic.go:356] "Generic (PLEG): container finished" podID="9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb" containerID="5f7ae203531df58e871f72742ddf66bd6dc725c8937057fb86b814df0d5fe9b1" exitCode=0
Sep 17 16:48:00.401953 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:00.401427    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v7rc/must-gather-8qb9d" event={"ID":"9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb","Type":"ContainerDied","Data":"5f7ae203531df58e871f72742ddf66bd6dc725c8937057fb86b814df0d5fe9b1"}
Sep 17 16:48:00.401953 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:00.401867    2574 scope.go:117] "RemoveContainer" containerID="5f7ae203531df58e871f72742ddf66bd6dc725c8937057fb86b814df0d5fe9b1"
Sep 17 16:48:00.881222 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:00.881182    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7v7rc_must-gather-8qb9d_9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb/gather/0.log"
Sep 17 16:48:01.344495 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.344378    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:01.503360 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.503320    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:01.512819 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.512789    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-przwd/must-gather-2dfxl"]
Sep 17 16:48:01.516983 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.516964    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-przwd/must-gather-2dfxl"
Sep 17 16:48:01.519858 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.519839    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-przwd\"/\"kube-root-ca.crt\""
Sep 17 16:48:01.520377 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.520363    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-przwd\"/\"default-dockercfg-gx5hg\""
Sep 17 16:48:01.520415 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.520402    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-przwd\"/\"openshift-service-ca.crt\""
Sep 17 16:48:01.528676 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.528647    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-przwd/must-gather-2dfxl"]
Sep 17 16:48:01.606800 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.606766    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:01.652944 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.652896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj744\" (UniqueName: \"kubernetes.io/projected/dcac0c44-fcc9-4d30-a9cd-2a487c890865-kube-api-access-gj744\") pod \"must-gather-2dfxl\" (UID: \"dcac0c44-fcc9-4d30-a9cd-2a487c890865\") " pod="openshift-must-gather-przwd/must-gather-2dfxl"
Sep 17 16:48:01.653112 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.652978    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dcac0c44-fcc9-4d30-a9cd-2a487c890865-must-gather-output\") pod \"must-gather-2dfxl\" (UID: \"dcac0c44-fcc9-4d30-a9cd-2a487c890865\") " pod="openshift-must-gather-przwd/must-gather-2dfxl"
Sep 17 16:48:01.754308 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.754256    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj744\" (UniqueName: \"kubernetes.io/projected/dcac0c44-fcc9-4d30-a9cd-2a487c890865-kube-api-access-gj744\") pod \"must-gather-2dfxl\" (UID: \"dcac0c44-fcc9-4d30-a9cd-2a487c890865\") " pod="openshift-must-gather-przwd/must-gather-2dfxl"
Sep 17 16:48:01.754520 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.754361    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dcac0c44-fcc9-4d30-a9cd-2a487c890865-must-gather-output\") pod \"must-gather-2dfxl\" (UID: \"dcac0c44-fcc9-4d30-a9cd-2a487c890865\") " pod="openshift-must-gather-przwd/must-gather-2dfxl"
Sep 17 16:48:01.754729 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.754712    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dcac0c44-fcc9-4d30-a9cd-2a487c890865-must-gather-output\") pod \"must-gather-2dfxl\" (UID: \"dcac0c44-fcc9-4d30-a9cd-2a487c890865\") " pod="openshift-must-gather-przwd/must-gather-2dfxl"
Sep 17 16:48:01.763682 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.763661    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj744\" (UniqueName: \"kubernetes.io/projected/dcac0c44-fcc9-4d30-a9cd-2a487c890865-kube-api-access-gj744\") pod \"must-gather-2dfxl\" (UID: \"dcac0c44-fcc9-4d30-a9cd-2a487c890865\") " pod="openshift-must-gather-przwd/must-gather-2dfxl"
Sep 17 16:48:01.826771 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:01.826726    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-przwd/must-gather-2dfxl"
Sep 17 16:48:02.113725 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:02.113598    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-przwd/must-gather-2dfxl"]
Sep 17 16:48:02.116612 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:48:02.116568    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcac0c44_fcc9_4d30_a9cd_2a487c890865.slice/crio-71aa4c45ddca1ae576eca7ee66bda1616606c2849c10d3d66b840d0569fdd686 WatchSource:0}: Error finding container 71aa4c45ddca1ae576eca7ee66bda1616606c2849c10d3d66b840d0569fdd686: Status 404 returned error can't find the container with id 71aa4c45ddca1ae576eca7ee66bda1616606c2849c10d3d66b840d0569fdd686
Sep 17 16:48:02.412039 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:02.412001    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-przwd/must-gather-2dfxl" event={"ID":"dcac0c44-fcc9-4d30-a9cd-2a487c890865","Type":"ContainerStarted","Data":"71aa4c45ddca1ae576eca7ee66bda1616606c2849c10d3d66b840d0569fdd686"}
Sep 17 16:48:03.420915 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:03.420825    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-przwd/must-gather-2dfxl" event={"ID":"dcac0c44-fcc9-4d30-a9cd-2a487c890865","Type":"ContainerStarted","Data":"9d9260c94112f1bd011d4e4d2734a3ee735dbd01dcf33f0c83af5f6c60bb88cb"}
Sep 17 16:48:03.420915 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:03.420870    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-przwd/must-gather-2dfxl" event={"ID":"dcac0c44-fcc9-4d30-a9cd-2a487c890865","Type":"ContainerStarted","Data":"9de5c77d37ea060592eb2909fdd18c0a040b9112daf9e9e69681aa3974d593f0"}
Sep 17 16:48:03.440102 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:03.440038    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-przwd/must-gather-2dfxl" podStartSLOduration=1.3924810060000001 podStartE2EDuration="2.44001658s" totalImagesPullingTime="1.047535574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:48:03.438640066 +0000 UTC m=+4148.910479924" watchObservedRunningTime="2026-09-17 16:48:03.44001658 +0000 UTC m=+4148.911856430"
Sep 17 16:48:04.940405 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:04.940367    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vt2l8_72f72846-2cc4-4793-a883-42382a787e30/global-pull-secret-syncer/0.log"
Sep 17 16:48:05.023901 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:05.023866    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log"
Sep 17 16:48:05.145972 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:05.145939    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-37.ec2.internal_6abc4d8c8ce3d693c97b0f59444f7f92/haproxy/0.log"
Sep 17 16:48:06.348712 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:06.348674    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-7v7rc/must-gather-8qb9d"]
Sep 17 16:48:06.349274 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:06.348941    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-7v7rc/must-gather-8qb9d" podUID="9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb" containerName="copy" containerID="cri-o://691d3c304c773acbc646c25e4005a707fbe6603e2f693a1cf57ebd55fe20f3fa" gracePeriod=2
Sep 17 16:48:06.357478 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:06.357322    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-7v7rc/must-gather-8qb9d"]
Sep 17 16:48:06.752462 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:06.751784    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7v7rc_must-gather-8qb9d_9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb/copy/0.log"
Sep 17 16:48:06.756460 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:06.754230    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v7rc/must-gather-8qb9d"
Sep 17 16:48:06.759489 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:48:06.758137    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8qb9d\" is forbidden: User \"system:node:ip-10-0-129-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7v7rc\": no relationship found between node 'ip-10-0-129-37.ec2.internal' and this object" podUID="9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb" pod="openshift-must-gather-7v7rc/must-gather-8qb9d"
Sep 17 16:48:06.831413 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:06.831364    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t758\" (UniqueName: \"kubernetes.io/projected/9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb-kube-api-access-6t758\") pod \"9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb\" (UID: \"9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb\") "
Sep 17 16:48:06.831651 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:06.831571    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb-must-gather-output\") pod \"9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb\" (UID: \"9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb\") "
Sep 17 16:48:06.841508 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:06.840823    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb" (UID: "9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:48:06.842930 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:06.842172    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb-kube-api-access-6t758" (OuterVolumeSpecName: "kube-api-access-6t758") pod "9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb" (UID: "9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb"). InnerVolumeSpecName "kube-api-access-6t758". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:48:06.932619 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:06.932573    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t758\" (UniqueName: \"kubernetes.io/projected/9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb-kube-api-access-6t758\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:48:06.932619 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:06.932619    2574 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb-must-gather-output\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 17 16:48:07.052862 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:07.052774    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb" path="/var/lib/kubelet/pods/9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb/volumes"
Sep 17 16:48:07.450559 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:07.450527    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7v7rc_must-gather-8qb9d_9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb/copy/0.log"
Sep 17 16:48:07.451647 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:07.451613    2574 generic.go:356] "Generic (PLEG): container finished" podID="9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb" containerID="691d3c304c773acbc646c25e4005a707fbe6603e2f693a1cf57ebd55fe20f3fa" exitCode=143
Sep 17 16:48:07.451811 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:07.451798    2574 scope.go:117] "RemoveContainer" containerID="691d3c304c773acbc646c25e4005a707fbe6603e2f693a1cf57ebd55fe20f3fa"
Sep 17 16:48:07.451979 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:07.451964    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v7rc/must-gather-8qb9d"
Sep 17 16:48:07.465207 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:07.465176    2574 scope.go:117] "RemoveContainer" containerID="5f7ae203531df58e871f72742ddf66bd6dc725c8937057fb86b814df0d5fe9b1"
Sep 17 16:48:07.497770 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:07.497737    2574 scope.go:117] "RemoveContainer" containerID="691d3c304c773acbc646c25e4005a707fbe6603e2f693a1cf57ebd55fe20f3fa"
Sep 17 16:48:07.499160 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:48:07.499121    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691d3c304c773acbc646c25e4005a707fbe6603e2f693a1cf57ebd55fe20f3fa\": container with ID starting with 691d3c304c773acbc646c25e4005a707fbe6603e2f693a1cf57ebd55fe20f3fa not found: ID does not exist" containerID="691d3c304c773acbc646c25e4005a707fbe6603e2f693a1cf57ebd55fe20f3fa"
Sep 17 16:48:07.499285 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:07.499182    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691d3c304c773acbc646c25e4005a707fbe6603e2f693a1cf57ebd55fe20f3fa"} err="failed to get container status \"691d3c304c773acbc646c25e4005a707fbe6603e2f693a1cf57ebd55fe20f3fa\": rpc error: code = NotFound desc = could not find container \"691d3c304c773acbc646c25e4005a707fbe6603e2f693a1cf57ebd55fe20f3fa\": container with ID starting with 691d3c304c773acbc646c25e4005a707fbe6603e2f693a1cf57ebd55fe20f3fa not found: ID does not exist"
Sep 17 16:48:07.499285 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:07.499211    2574 scope.go:117] "RemoveContainer" containerID="5f7ae203531df58e871f72742ddf66bd6dc725c8937057fb86b814df0d5fe9b1"
Sep 17 16:48:07.499641 ip-10-0-129-37 kubenswrapper[2574]: E0917 16:48:07.499614    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7ae203531df58e871f72742ddf66bd6dc725c8937057fb86b814df0d5fe9b1\": container with ID starting with 5f7ae203531df58e871f72742ddf66bd6dc725c8937057fb86b814df0d5fe9b1 not found: ID does not exist" containerID="5f7ae203531df58e871f72742ddf66bd6dc725c8937057fb86b814df0d5fe9b1"
Sep 17 16:48:07.499750 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:07.499672    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7ae203531df58e871f72742ddf66bd6dc725c8937057fb86b814df0d5fe9b1"} err="failed to get container status \"5f7ae203531df58e871f72742ddf66bd6dc725c8937057fb86b814df0d5fe9b1\": rpc error: code = NotFound desc = could not find container \"5f7ae203531df58e871f72742ddf66bd6dc725c8937057fb86b814df0d5fe9b1\": container with ID starting with 5f7ae203531df58e871f72742ddf66bd6dc725c8937057fb86b814df0d5fe9b1 not found: ID does not exist"
Sep 17 16:48:08.398098 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:08.398042    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:08.499757 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:08.499718    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t4xp_c35b1dbc-5400-46a6-a74a-1fac21cf6cff/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:09.872691 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:09.872656    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-26g8k_d06c1525-0f0c-4b12-b6ef-9b2a297400af/manager/0.log"
Sep 17 16:48:10.086246 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:10.086207    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-szvjq_ca00977e-3c20-4628-ae3d-125aa5d1756e/manager/0.log"
Sep 17 16:48:10.168072 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:10.167979    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-892hx_1facedd8-3eae-45ce-9516-21618b921ea5/limitador/0.log"
Sep 17 16:48:10.243012 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:10.242977    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qxkqk_e5f51b66-517d-4073-82ad-95511a0cca5d/manager/0.log"
Sep 17 16:48:11.593681 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:11.593652    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_71c12437-d48a-463a-a015-337c239986ac/alertmanager/0.log"
Sep 17 16:48:11.622468 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:11.622409    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_71c12437-d48a-463a-a015-337c239986ac/config-reloader/0.log"
Sep 17 16:48:11.657221 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:11.657173    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_71c12437-d48a-463a-a015-337c239986ac/kube-rbac-proxy-web/0.log"
Sep 17 16:48:11.680748 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:11.680714    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_71c12437-d48a-463a-a015-337c239986ac/kube-rbac-proxy/0.log"
Sep 17 16:48:11.708598 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:11.708523    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_71c12437-d48a-463a-a015-337c239986ac/kube-rbac-proxy-metric/0.log"
Sep 17 16:48:11.743311 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:11.743281    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_71c12437-d48a-463a-a015-337c239986ac/prom-label-proxy/0.log"
Sep 17 16:48:11.779154 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:11.779119    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_71c12437-d48a-463a-a015-337c239986ac/init-config-reloader/0.log"
Sep 17 16:48:11.992083 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:11.992033    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7f9446879-llkrn_1191d1d7-63b6-43db-9065-9103a0195353/metrics-server/0.log"
Sep 17 16:48:12.250421 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:12.250336    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rnt5p_25f51d67-a0da-4d03-8fa9-9149fcdbbb92/node-exporter/0.log"
Sep 17 16:48:12.301387 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:12.301348    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rnt5p_25f51d67-a0da-4d03-8fa9-9149fcdbbb92/kube-rbac-proxy/0.log"
Sep 17 16:48:12.368460 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:12.368028    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rnt5p_25f51d67-a0da-4d03-8fa9-9149fcdbbb92/init-textfile/0.log"
Sep 17 16:48:13.066071 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:13.066035    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-qbc2c_f463991b-2e30-48fb-9007-7a6a4fa6a001/prometheus-operator/0.log"
Sep 17 16:48:13.095308 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:13.095273    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-qbc2c_f463991b-2e30-48fb-9007-7a6a4fa6a001/kube-rbac-proxy/0.log"
Sep 17 16:48:13.205456 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:13.205408    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c7dcb6fd8-s2rht_b6f79929-5d0a-4e50-898d-aac097380652/telemeter-client/0.log"
Sep 17 16:48:13.250359 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:13.250324    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c7dcb6fd8-s2rht_b6f79929-5d0a-4e50-898d-aac097380652/reload/0.log"
Sep 17 16:48:13.285998 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:13.285964    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c7dcb6fd8-s2rht_b6f79929-5d0a-4e50-898d-aac097380652/kube-rbac-proxy/0.log"
Sep 17 16:48:13.331664 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:13.331569    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7948667c77-xgmc9_3925522f-d646-4bec-b1c1-58d3383380df/thanos-query/0.log"
Sep 17 16:48:13.372003 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:13.371970    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7948667c77-xgmc9_3925522f-d646-4bec-b1c1-58d3383380df/kube-rbac-proxy-web/0.log"
Sep 17 16:48:13.412293 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:13.412258    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7948667c77-xgmc9_3925522f-d646-4bec-b1c1-58d3383380df/kube-rbac-proxy/0.log"
Sep 17 16:48:13.446083 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:13.446054    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7948667c77-xgmc9_3925522f-d646-4bec-b1c1-58d3383380df/prom-label-proxy/0.log"
Sep 17 16:48:13.481813 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:13.481780    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7948667c77-xgmc9_3925522f-d646-4bec-b1c1-58d3383380df/kube-rbac-proxy-rules/0.log"
Sep 17 16:48:13.526024 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:13.525988    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7948667c77-xgmc9_3925522f-d646-4bec-b1c1-58d3383380df/kube-rbac-proxy-metrics/0.log"
Sep 17 16:48:14.917041 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:14.917007    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"]
Sep 17 16:48:14.917642 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:14.917620    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb" containerName="copy"
Sep 17 16:48:14.917731 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:14.917645    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb" containerName="copy"
Sep 17 16:48:14.917731 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:14.917681    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb" containerName="gather"
Sep 17 16:48:14.917731 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:14.917690    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb" containerName="gather"
Sep 17 16:48:14.917885 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:14.917813    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb" containerName="gather"
Sep 17 16:48:14.917885 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:14.917829    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1a8772-e8f4-4bd9-9bdb-c9a1f4504dbb" containerName="copy"
Sep 17 16:48:14.922934 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:14.922896    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:14.941580 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:14.941551    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"]
Sep 17 16:48:15.019468 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.019418    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00f5adf8-3a88-4189-ab40-e0bff082a455-sys\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.019639 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.019507    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/00f5adf8-3a88-4189-ab40-e0bff082a455-proc\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.019639 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.019527    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/00f5adf8-3a88-4189-ab40-e0bff082a455-podres\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.019639 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.019567    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrmvm\" (UniqueName: \"kubernetes.io/projected/00f5adf8-3a88-4189-ab40-e0bff082a455-kube-api-access-qrmvm\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.019750 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.019649    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00f5adf8-3a88-4189-ab40-e0bff082a455-lib-modules\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.121300 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.121245    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00f5adf8-3a88-4189-ab40-e0bff082a455-sys\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.121532 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.121386    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/00f5adf8-3a88-4189-ab40-e0bff082a455-proc\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.121532 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.121420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/00f5adf8-3a88-4189-ab40-e0bff082a455-podres\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.121532 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.121504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrmvm\" (UniqueName: \"kubernetes.io/projected/00f5adf8-3a88-4189-ab40-e0bff082a455-kube-api-access-qrmvm\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.121726 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.121563    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00f5adf8-3a88-4189-ab40-e0bff082a455-lib-modules\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.122510 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.122480    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00f5adf8-3a88-4189-ab40-e0bff082a455-sys\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.122912 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.122877    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/00f5adf8-3a88-4189-ab40-e0bff082a455-proc\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.123048 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.123009    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/00f5adf8-3a88-4189-ab40-e0bff082a455-podres\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.123170 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.123150    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00f5adf8-3a88-4189-ab40-e0bff082a455-lib-modules\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.132588 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.132551    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrmvm\" (UniqueName: \"kubernetes.io/projected/00f5adf8-3a88-4189-ab40-e0bff082a455-kube-api-access-qrmvm\") pod \"perf-node-gather-daemonset-nhfx7\" (UID: \"00f5adf8-3a88-4189-ab40-e0bff082a455\") " pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.236331 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.236019    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:15.546928 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:15.546896    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"]
Sep 17 16:48:15.550518 ip-10-0-129-37 kubenswrapper[2574]: W0917 16:48:15.550473    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod00f5adf8_3a88_4189_ab40_e0bff082a455.slice/crio-86a0052374ccc7462277ada60fd82a1d06471c0a302aa549a513b3de501f11b9 WatchSource:0}: Error finding container 86a0052374ccc7462277ada60fd82a1d06471c0a302aa549a513b3de501f11b9: Status 404 returned error can't find the container with id 86a0052374ccc7462277ada60fd82a1d06471c0a302aa549a513b3de501f11b9
Sep 17 16:48:16.053263 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:16.053203    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6bc88b544-h4t9m_735788a4-f575-4ddb-b1a6-c8c0cd80d7f6/console/0.log"
Sep 17 16:48:16.519749 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:16.519710    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7" event={"ID":"00f5adf8-3a88-4189-ab40-e0bff082a455","Type":"ContainerStarted","Data":"14f2ee14b7f6598198ed94443fb617c234ed9c41deaa6003725bca1bffc182aa"}
Sep 17 16:48:16.519978 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:16.519800    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7" event={"ID":"00f5adf8-3a88-4189-ab40-e0bff082a455","Type":"ContainerStarted","Data":"86a0052374ccc7462277ada60fd82a1d06471c0a302aa549a513b3de501f11b9"}
Sep 17 16:48:16.519978 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:16.519824    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:16.540312 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:16.540233    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7" podStartSLOduration=2.540209773 podStartE2EDuration="2.540209773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:48:16.538414307 +0000 UTC m=+4162.010254154" watchObservedRunningTime="2026-09-17 16:48:16.540209773 +0000 UTC m=+4162.012049620"
Sep 17 16:48:17.839490 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:17.839461    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d7xgf_300fa6c7-126b-4872-bb0f-13ca5a4eca5c/dns/0.log"
Sep 17 16:48:17.862351 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:17.862317    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d7xgf_300fa6c7-126b-4872-bb0f-13ca5a4eca5c/kube-rbac-proxy/0.log"
Sep 17 16:48:18.067343 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:18.067320    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vzk9l_9f8f3b74-61b5-4b22-bf59-ab2ec9054df1/dns-node-resolver/0.log"
Sep 17 16:48:18.885083 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:18.885049    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7845q_346c8e4c-4119-41a8-b798-c23d6de9b203/node-ca/0.log"
Sep 17 16:48:20.099010 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:20.098969    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64b88459dd-xdnj5_1d1c6522-62ea-4c17-b992-33f236073899/router/0.log"
Sep 17 16:48:20.753870 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:20.753803    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lpgls_02f7fe3e-05c4-49ac-97d2-35337336c9cb/serve-healthcheck-canary/0.log"
Sep 17 16:48:21.408406 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:21.408368    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7sgh7_117720a7-6661-4079-81b4-f5b65382cfb6/kube-rbac-proxy/0.log"
Sep 17 16:48:21.436020 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:21.435989    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7sgh7_117720a7-6661-4079-81b4-f5b65382cfb6/exporter/0.log"
Sep 17 16:48:21.461182 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:21.461152    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7sgh7_117720a7-6661-4079-81b4-f5b65382cfb6/extractor/0.log"
Sep 17 16:48:22.537418 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:22.537375    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-przwd/perf-node-gather-daemonset-nhfx7"
Sep 17 16:48:25.067185 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:25.067152    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64b88459dd-xdnj5_1d1c6522-62ea-4c17-b992-33f236073899/router/0.log"
Sep 17 16:48:25.965820 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:25.965787    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-74b7f9c78b-78vlf_299537d9-a2d7-4dd2-ae91-1e72e66b69f7/manager/0.log"
Sep 17 16:48:26.229677 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:26.229600    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64b88459dd-xdnj5_1d1c6522-62ea-4c17-b992-33f236073899/router/0.log"
Sep 17 16:48:26.456553 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:26.456515    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-745575db98-qjzqx_b708ab85-a7dd-478d-af39-33b54278e0f7/manager/0.log"
Sep 17 16:48:26.511664 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:26.511590    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-47xz5_e281b834-ad84-4766-be56-4a9626f6a1e1/server/0.log"
Sep 17 16:48:27.622504 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:27.622466    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-5x6jm_135c2239-af06-4865-ada4-4414462f4965/manager/0.log"
Sep 17 16:48:27.705365 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:27.705280    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qhnrx_10dadd7b-9637-45d1-b83a-c2d8180cf080/s3-init/0.log"
Sep 17 16:48:27.817714 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:27.817683    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qhnrx_10dadd7b-9637-45d1-b83a-c2d8180cf080/download-hf-model/0.log"
Sep 17 16:48:27.897541 ip-10-0-129-37 kubenswrapper[2574]: I0917 16:48:27.897505    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-2dgms_d272397f-264e-443b-9298-ee3a3ecbd9ef/seaweedfs/0.log"