Sep 17 18:37:08.326119 ip-10-0-140-194 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 18:37:08.326134 ip-10-0-140-194 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 18:37:08.326145 ip-10-0-140-194 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 18:37:08.326519 ip-10-0-140-194 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 18:37:18.521071 ip-10-0-140-194 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 18:37:18.521086 ip-10-0-140-194 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1c2a82f98d2a4760b610d3ff6c753c3e --
Sep 17 18:39:06.657306 ip-10-0-140-194 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 18:39:10.042560 ip-10-0-140-194 kubenswrapper[2587]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:39:10.042560 ip-10-0-140-194 kubenswrapper[2587]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 18:39:10.042560 ip-10-0-140-194 kubenswrapper[2587]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:39:10.042560 ip-10-0-140-194 kubenswrapper[2587]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 18:39:10.042560 ip-10-0-140-194 kubenswrapper[2587]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:39:10.043300 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.042626    2587 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 18:39:10.046555 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046535    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:10.046555 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046554    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046558    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046562    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046565    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046568    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046573    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046576    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046579    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046583    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046587    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046590    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046593    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046595    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046598    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046601    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046603    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046606    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046609    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046612    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046614    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:10.046623 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046617    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046619    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046622    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046625    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046629    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046634    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046637    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046640    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046643    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046646    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046649    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046652    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046654    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046657    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046660    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046663    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046666    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046669    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046671    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046674    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:10.047167 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046676    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046679    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046681    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046683    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046686    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046689    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046691    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046694    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046696    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046699    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046701    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046704    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046706    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046709    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046711    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046714    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046717    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046720    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046722    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:10.047650 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046725    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046727    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046730    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046732    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046735    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046738    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046741    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046744    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046747    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046749    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046752    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046755    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046758    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046760    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046764    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046766    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046769    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046771    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046773    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046777    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:10.048129 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046781    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046784    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046787    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046789    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046792    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046795    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046797    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046803    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046806    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046809    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046812    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046815    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046818    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046821    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046824    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046826    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046829    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046831    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.046834    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047551    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:10.048605 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047558    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047562    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047566    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047569    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047571    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047574    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047577    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047580    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047583    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047586    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047588    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047591    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047594    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047600    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047603    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047606    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047608    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047611    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047613    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047616    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:10.049438 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047618    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047621    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047623    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047626    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047628    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047631    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047633    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047636    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047638    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047641    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047643    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047646    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047648    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047651    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047654    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047657    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047660    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047662    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047665    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:10.050297 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047668    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047670    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047676    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047679    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047682    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047685    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047688    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047690    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047693    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047695    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047698    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047700    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047703    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047706    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047708    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047711    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047713    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047717    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047719    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047722    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:10.051114 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047724    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047727    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047729    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047731    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047734    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047737    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047739    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047742    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047746    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047748    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047751    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047755    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047759    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047761    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047764    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047767    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047770    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047774    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047778    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:10.051601 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047782    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047784    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047787    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047790    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047793    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047795    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047798    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047801    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047804    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047806    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047809    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047811    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047814    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047816    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047818    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047821    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047823    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047826    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047828    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:10.052183 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.047831    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048260    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048267    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048277    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048281    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048284    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048287    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048290    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048293    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048296    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048299    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048301    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048304    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048307    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048309    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048312    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048314    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048318    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048321    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:10.052680 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048324    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048327    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048330    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048333    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048336    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048339    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048342    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048344    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048347    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048349    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048352    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048355    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048357    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048361    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048366    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048370    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048373    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048376    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048379    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048381    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:10.053263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048384    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048387    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048389    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048391    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048394    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048396    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048398    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048401    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048403    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048406    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048408    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048411    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048413    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048415    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048418    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048420    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048423    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048425    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048428    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048430    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:10.053870 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048432    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048435    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048437    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048440    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048442    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048446    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048449    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048452    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048455    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048458    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048461    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048463    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048466    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048469    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048471    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048474    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048476    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048479    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048481    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:10.054414 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048484    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048486    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048488    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048491    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048495    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048497    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048500    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048502    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048505    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048507    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048510    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048515    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048517    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048520    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048522    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048525    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048527    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048530    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048532    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048535    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:10.054963 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048537    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.048539    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.048610    2587 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.048685    2587 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.048728    2587 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049231    2587 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049257    2587 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049265    2587 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049274    2587 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049282    2587 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049289    2587 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049295    2587 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049301    2587 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049315    2587 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049320    2587 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049326    2587 flags.go:64] FLAG: --cgroup-root=""
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049331    2587 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049336    2587 flags.go:64] FLAG: --client-ca-file=""
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049341    2587 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049346    2587 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049366    2587 flags.go:64] FLAG: --cluster-domain=""
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049371    2587 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049381    2587 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049386    2587 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 18:39:10.055461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049391    2587 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049398    2587 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049403    2587 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049408    2587 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049414    2587 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049419    2587 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049429    2587 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049435    2587 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049441    2587 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049446    2587 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049453    2587 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049458    2587 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049463    2587 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049468    2587 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049479    2587 flags.go:64] FLAG: --enable-server="true"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049484    2587 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049493    2587 flags.go:64] FLAG: --event-burst="100"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049498    2587 flags.go:64] FLAG: --event-qps="50"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049503    2587 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049507    2587 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049512    2587 flags.go:64] FLAG: --eviction-hard=""
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049519    2587 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049523    2587 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049533    2587 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049539    2587 flags.go:64] FLAG: --eviction-soft=""
Sep 17 18:39:10.056068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049543    2587 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049549    2587 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049554    2587 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049558    2587 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049563    2587 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049568    2587 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049577    2587 flags.go:64] FLAG: --feature-gates=""
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049584    2587 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049589    2587 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049594    2587 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049599    2587 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049607    2587 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049612    2587 flags.go:64] FLAG: --help="false"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049617    2587 flags.go:64] FLAG: --hostname-override="ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049622    2587 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049633    2587 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049638    2587 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049644    2587 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049652    2587 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049656    2587 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049661    2587 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049667    2587 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049672    2587 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 18:39:10.056712 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049681    2587 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049687    2587 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049691    2587 flags.go:64] FLAG: --kube-reserved=""
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049696    2587 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049701    2587 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049706    2587 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049711    2587 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049716    2587 flags.go:64] FLAG: --lock-file=""
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049720    2587 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049729    2587 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049734    2587 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049743    2587 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049748    2587 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049752    2587 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049757    2587 flags.go:64] FLAG: --logging-format="text"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049762    2587 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049768    2587 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049778    2587 flags.go:64] FLAG: --manifest-url=""
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049782    2587 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049789    2587 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049794    2587 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049800    2587 flags.go:64] FLAG: --max-pods="110"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049805    2587 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049810    2587 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049829    2587 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 18:39:10.057308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.049875    2587 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050048    2587 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050055    2587 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050061    2587 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050077    2587 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050083    2587 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050088    2587 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050093    2587 flags.go:64] FLAG: --pod-cidr=""
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050098    2587 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050106    2587 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050111    2587 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050117    2587 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050121    2587 flags.go:64] FLAG: --port="10250"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050126    2587 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050131    2587 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-066e5e818efc445f8"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050137    2587 flags.go:64] FLAG: --qos-reserved=""
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050142    2587 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050147    2587 flags.go:64] FLAG: --register-node="true"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050152    2587 flags.go:64] FLAG: --register-with-taints=""
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050158    2587 flags.go:64] FLAG: --registry-burst="10"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050162    2587 flags.go:64] FLAG: --registry-qps="5"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050167    2587 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050172    2587 flags.go:64] FLAG: --reserved-memory=""
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050177    2587 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050183    2587 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 18:39:10.058002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050188    2587 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050192    2587 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050197    2587 flags.go:64] FLAG: --runonce="false"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050201    2587 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050206    2587 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050211    2587 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050217    2587 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050221    2587 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050227    2587 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050233    2587 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050238    2587 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050242    2587 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050247    2587 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050252    2587 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050256    2587 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050261    2587 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050266    2587 flags.go:64] FLAG: --system-cgroups=""
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050271    2587 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050280    2587 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050284    2587 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050289    2587 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050297    2587 flags.go:64] FLAG: --tls-min-version=""
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050301    2587 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050305    2587 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050309    2587 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 18:39:10.058655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050313    2587 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050318    2587 flags.go:64] FLAG: --v="2"
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050324    2587 flags.go:64] FLAG: --version="false"
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050331    2587 flags.go:64] FLAG: --vmodule=""
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050346    2587 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.050351    2587 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050544    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050551    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050556    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050560    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050566    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050571    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050576    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050581    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050586    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050594    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050598    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050603    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050607    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050612    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050616    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:10.059301 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050620    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050624    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050629    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050633    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050637    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050641    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050645    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050649    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050653    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050658    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050662    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050666    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050670    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050674    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050679    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050683    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050687    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050691    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050696    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050700    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:10.059810 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050704    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050709    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050713    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050717    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050721    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050725    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050732    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050736    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050741    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050745    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050749    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050753    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050758    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050762    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050766    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050770    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050774    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050778    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050782    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050787    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:10.060303 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050791    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050795    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050802    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050809    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050813    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050819    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050825    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050830    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050835    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050839    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050845    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050850    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050855    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050859    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050863    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050870    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050877    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050881    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:10.060798 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050888    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050894    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050898    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050902    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050907    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050911    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050915    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050920    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050924    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050928    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050932    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050957    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050962    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050967    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050971    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050975    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050979    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050983    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050988    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050992    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:10.061263 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.050996    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.051000    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.051004    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.051008    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.051012    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.051017    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.051674    2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.059603    2587 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.059619    2587 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.059642    2587 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.059648    2587 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059708    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059713    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059716    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059719    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:10.061739 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059722    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059726    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059729    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059731    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059734    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059737    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059739    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059742    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059745    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059747    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059750    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059753    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059755    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059758    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059760    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059763    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059765    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059768    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059770    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059773    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:10.062127 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059776    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059778    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059781    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059784    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059786    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059791    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059794    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059797    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059800    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059802    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059805    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059808    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059810    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059813    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059816    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059819    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059821    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059824    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059827    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059830    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:10.062600 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059834    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059838    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059842    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059846    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059849    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059851    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059854    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059856    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059859    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059862    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059865    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059867    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059870    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059872    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059875    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059878    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059880    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059884    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059887    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059890    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:10.063262 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059893    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059896    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059898    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059901    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059903    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059906    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059909    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059911    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059914    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059917    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059919    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059930    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059934    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059953    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059957    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059960    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059963    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059966    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059968    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059971    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:10.063780 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059973    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:10.064282 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059977    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:10.064282 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059979    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:10.064282 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059982    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:10.064282 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059985    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:10.064282 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059987    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:10.064282 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059990    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:10.064282 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059994    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:10.064282 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059996    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:10.064282 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.059999    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:10.064282 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060002    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:10.064282 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060005    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:10.064282 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060008    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:10.064282 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060012    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:10.064282 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060016    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.060021    2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060156    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060161    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060164    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060167    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060170    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060174    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060177    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060179    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060182    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060185    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060188    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060190    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060193    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:39:10.064628 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060195    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060198    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060201    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060203    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060206    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060209    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060211    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060214    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060217    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060219    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060223    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060227    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060230    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060233    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060236    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060239    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060243    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060246    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060249    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:39:10.065012 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060252    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060255    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060257    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060260    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060263    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060266    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060268    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060271    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060274    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060277    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060279    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060282    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060284    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060287    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060290    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060293    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060295    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060298    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060300    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060303    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:39:10.065459 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060305    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060308    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060311    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060313    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060316    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060319    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060321    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060324    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060327    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060330    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060332    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060335    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060337    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060340    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060343    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060345    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060348    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060352    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060355    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:39:10.065926 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060358    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060361    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060363    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060366    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060368    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060371    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060374    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060376    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060379    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060381    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060384    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060386    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060389    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060391    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060394    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060396    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060399    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060401    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060404    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060407    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:39:10.066404 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060409    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:39:10.066884 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060412    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:39:10.066884 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060414    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:39:10.066884 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060418    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:39:10.066884 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060421    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:39:10.066884 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060423    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:39:10.066884 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060426    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:39:10.066884 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:10.060428    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:39:10.066884 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.060433    2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:39:10.066884 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.060577    2587 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 18:39:10.066884 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.062814    2587 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 18:39:10.066884 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.063783    2587 server.go:1019] "Starting client certificate rotation"
Sep 17 18:39:10.066884 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.063884    2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:39:10.066884 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.064698    2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:39:10.067235 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.066395    2587 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:39:10.067573 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.067558    2587 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:39:10.074044 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.074024    2587 log.go:25] "Validated CRI v1 runtime API"
Sep 17 18:39:10.076862 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.076769    2587 log.go:25] "Validated CRI v1 image API"
Sep 17 18:39:10.077919 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.077903    2587 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 18:39:10.080727 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.080674    2587 fs.go:135] Filesystem UUIDs: map[6d9554e7-9094-466c-b8cf-73ab27381f16:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 974a061f-533d-409f-b0cf-07d097949576:/dev/nvme0n1p3]
Sep 17 18:39:10.080863 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.080721    2587 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 18:39:10.087636 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.087499    2587 manager.go:216] Machine: {Timestamp:2026-09-17 18:39:10.085435772 +0000 UTC m=+1.964404521 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092761 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20f4ee0e4bc3a7cc1d6e49921736b1 SystemUUID:ec20f4ee-0e4b-c3a7-cc1d-6e49921736b1 BootID:1c2a82f9-8d2a-4760-b610-d3ff6c753c3e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fc:c0:e4:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fc:c0:e4:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:f8:7d:ec:cf:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 18:39:10.087636 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.087617    2587 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 18:39:10.087798 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.087718    2587 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 18:39:10.089399 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.089369    2587 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 18:39:10.089576 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.089401    2587 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 18:39:10.090673 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.090662    2587 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 18:39:10.090713 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.090676    2587 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 18:39:10.090713 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.090684    2587 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:39:10.090713 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.090703    2587 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:39:10.090794 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.090788    2587 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 18:39:10.094502 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.094486    2587 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:39:10.094640 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.094630    2587 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 18:39:10.094672 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.094665    2587 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 18:39:10.094699 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.094678    2587 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 18:39:10.094699 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.094693    2587 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 18:39:10.094757 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.094703    2587 kubelet.go:398] "Adding apiserver pod source"
Sep 17 18:39:10.094757 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.094712    2587 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 18:39:10.103529 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.103510    2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:39:10.103529 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.103535    2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:39:10.103702 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.103556    2587 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:39:10.105656 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.105628    2587 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 18:39:10.107221 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.107200    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 18:39:10.107282 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.107199    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 18:39:10.107822 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.107809    2587 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 18:39:10.107858 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.107840    2587 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 18:39:10.109046 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.109034    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 18:39:10.109090 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.109051    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 18:39:10.109090 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.109057    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 18:39:10.109090 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.109062    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 18:39:10.109090 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.109072    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 18:39:10.109090 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.109081    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 18:39:10.109220 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.109097    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 18:39:10.109220 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.109102    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 18:39:10.109220 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.109110    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 18:39:10.109220 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.109115    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 18:39:10.109220 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.109124    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 18:39:10.109220 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.109130    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 18:39:10.109220 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.109154    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 18:39:10.109220 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.109162    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 18:39:10.113007 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.112992    2587 server.go:1268] "Started kubelet"
Sep 17 18:39:10.113088 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.113051    2587 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 18:39:10.113711 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.113641    2587 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 18:39:10.113781 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.113735    2587 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 18:39:10.114588 ip-10-0-140-194 systemd[1]: Started Kubernetes Kubelet.
Sep 17 18:39:10.115590 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.114827    2587 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 18:39:10.115590 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.114985    2587 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 18:39:10.118519 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.118489    2587 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 18:39:10.120001 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.119987    2587 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 18:39:10.120075 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.119995    2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 18:39:10.120126 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.120113    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:10.120723 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.120705    2587 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 18:39:10.120723 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.120724    2587 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 18:39:10.120855 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.120741    2587 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 18:39:10.120855 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.120766    2587 factory.go:55] Registering systemd factory
Sep 17 18:39:10.120855 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.120781    2587 factory.go:223] Registration of the systemd container factory successfully
Sep 17 18:39:10.120981 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.120876    2587 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 18:39:10.120981 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.120885    2587 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 18:39:10.121064 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.121039    2587 factory.go:153] Registering CRI-O factory
Sep 17 18:39:10.121064 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.121054    2587 factory.go:223] Registration of the crio container factory successfully
Sep 17 18:39:10.121145 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.121121    2587 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 18:39:10.121145 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.121141    2587 factory.go:103] Registering Raw factory
Sep 17 18:39:10.121301 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.121155    2587 manager.go:1196] Started watching for new ooms in manager
Sep 17 18:39:10.121444 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.121440    2587 manager.go:349] Starting recovery of all containers
Sep 17 18:39:10.151215 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.151176    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 18:39:10.151367 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.151306    2587 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 18:39:10.151473 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.151453    2587 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 18:39:10.162255 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.162227    2587 manager.go:354] Recovery completed
Sep 17 18:39:10.166660 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.166643    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:10.168197 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.167302    2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-194.ec2.internal.18d62fb38b882fc1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-194.ec2.internal,UID:ip-10-0-140-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-194.ec2.internal,},FirstTimestamp:2026-09-17 18:39:10.112960449 +0000 UTC m=+1.991929197,LastTimestamp:2026-09-17 18:39:10.112960449 +0000 UTC m=+1.991929197,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-194.ec2.internal,}"
Sep 17 18:39:10.168615 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.168589    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:10.168666 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.168630    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:10.168666 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.168641    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:10.169258 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.169241    2587 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 18:39:10.169258 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.169257    2587 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 18:39:10.169359 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.169274    2587 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:39:10.169359 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.169244    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:10.170121 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.170105    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:10.170208 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.170131    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:10.170208 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.170146    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:10.170502 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.170486    2587 policy_none.go:49] "None policy: Start"
Sep 17 18:39:10.170550 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.170510    2587 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 18:39:10.170550 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.170523    2587 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 18:39:10.171573 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.171558    2587 policy_none.go:47] "Start"
Sep 17 18:39:10.172239 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.172223    2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2mvlq"
Sep 17 18:39:10.180410 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.180387    2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2mvlq"
Sep 17 18:39:10.182390 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.182276    2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-194.ec2.internal.18d62fb38ed96f28  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-194.ec2.internal,UID:ip-10-0-140-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-194.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-194.ec2.internal,},FirstTimestamp:2026-09-17 18:39:10.168616744 +0000 UTC m=+2.047585491,LastTimestamp:2026-09-17 18:39:10.168616744 +0000 UTC m=+2.047585491,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-194.ec2.internal,}"
Sep 17 18:39:10.211669 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.211644    2587 manager.go:337] "Starting Device Plugin manager"
Sep 17 18:39:10.211669 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.211687    2587 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 18:39:10.214377 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.211698    2587 server.go:87] "Starting device plugin registration server"
Sep 17 18:39:10.214377 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.212013    2587 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 18:39:10.214377 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.212039    2587 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 18:39:10.214377 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.212145    2587 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 18:39:10.214377 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.212237    2587 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 18:39:10.214377 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.212247    2587 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 18:39:10.214377 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.212749    2587 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 18:39:10.214377 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.212793    2587 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:10.250431 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.250402    2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 18:39:10.251877 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.251856    2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 18:39:10.252008 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.251910    2587 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 18:39:10.252008 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.251952    2587 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 18:39:10.252008 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.251997    2587 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 18:39:10.254248 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.254221    2587 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:10.313054 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.312971    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:10.313930 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.313906    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:10.314033 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.313952    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:10.314033 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.313970    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:10.314033 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.313991    2587 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.320926 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.320907    2587 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.321061 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.320935    2587 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-194.ec2.internal\": node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:10.352283 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.352226    2587 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-194.ec2.internal"]
Sep 17 18:39:10.352446 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.352326    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:10.353247 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.353231    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:10.353307 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.353263    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:10.353307 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.353275    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:10.354604 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.354591    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:10.354717 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.354703    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.354777 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.354736    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:10.355459 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.355442    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:10.355537 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.355479    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:10.355537 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.355489    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:10.355537 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.355444    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:10.355643 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.355549    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:10.355643 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.355561    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:10.356886 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.356867    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.356981 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.356911    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:10.357666 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.357650    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:10.357755 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.357682    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:10.357755 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.357693    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:10.379700 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.379671    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:10.383799 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.383781    2587 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-194.ec2.internal\" not found" node="ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.388312 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.388294    2587 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-194.ec2.internal\" not found" node="ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.422636 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.422601    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdc7ceb412f9c95a22d608979ff76c2f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-194.ec2.internal\" (UID: \"fdc7ceb412f9c95a22d608979ff76c2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.422636 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.422638    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ce1f0d6f55636400e257394b4e53616-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal\" (UID: \"9ce1f0d6f55636400e257394b4e53616\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.422800 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.422660    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ce1f0d6f55636400e257394b4e53616-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal\" (UID: \"9ce1f0d6f55636400e257394b4e53616\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.480583 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.480550    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:10.523071 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.523033    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdc7ceb412f9c95a22d608979ff76c2f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-194.ec2.internal\" (UID: \"fdc7ceb412f9c95a22d608979ff76c2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.523229 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.523070    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ce1f0d6f55636400e257394b4e53616-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal\" (UID: \"9ce1f0d6f55636400e257394b4e53616\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.523229 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.523099    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ce1f0d6f55636400e257394b4e53616-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal\" (UID: \"9ce1f0d6f55636400e257394b4e53616\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.523229 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.523135    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ce1f0d6f55636400e257394b4e53616-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal\" (UID: \"9ce1f0d6f55636400e257394b4e53616\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.523229 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.523141    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdc7ceb412f9c95a22d608979ff76c2f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-194.ec2.internal\" (UID: \"fdc7ceb412f9c95a22d608979ff76c2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.523229 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.523140    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ce1f0d6f55636400e257394b4e53616-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal\" (UID: \"9ce1f0d6f55636400e257394b4e53616\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.581409 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.581319    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:10.682083 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.682046    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:10.688224 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.688201    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.692669 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:10.692444    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:10.783040 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.782842    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:10.883417 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.883336    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:10.983966 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:10.983917    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:11.021637 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.021605    2587 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:11.063934 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.063894    2587 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 18:39:11.064373 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.064072    2587 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:39:11.064373 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.064102    2587 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:39:11.084247 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:11.084209    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:11.166304 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.166236    2587 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:11.184960 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:11.184907    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:11.185089 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.184923    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:34:10 +0000 UTC" deadline="2028-04-17 14:04:45.923151343 +0000 UTC"
Sep 17 18:39:11.185089 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.184985    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13867h25m34.73817124s"
Sep 17 18:39:11.273906 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.273865    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:39:11.275007 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.274989    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:39:11.275117 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.275021    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:39:11.275117 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.275035    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:39:11.286007 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:11.285984    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:11.287594 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:11.287573    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce1f0d6f55636400e257394b4e53616.slice/crio-b067b87c70e9df2f0b1a9c7f355aa696fdb5a899f3c56255adb26c55246c1893 WatchSource:0}: Error finding container b067b87c70e9df2f0b1a9c7f355aa696fdb5a899f3c56255adb26c55246c1893: Status 404 returned error can't find the container with id b067b87c70e9df2f0b1a9c7f355aa696fdb5a899f3c56255adb26c55246c1893
Sep 17 18:39:11.288138 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:11.288117    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc7ceb412f9c95a22d608979ff76c2f.slice/crio-fb9bf6e7c30be2e4855d8959860027211d6820c62151c56cc7897597962651fa WatchSource:0}: Error finding container fb9bf6e7c30be2e4855d8959860027211d6820c62151c56cc7897597962651fa: Status 404 returned error can't find the container with id fb9bf6e7c30be2e4855d8959860027211d6820c62151c56cc7897597962651fa
Sep 17 18:39:11.294399 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.294380    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:39:11.386473 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:11.386416    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:11.487073 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:11.486982    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-194.ec2.internal\" not found"
Sep 17 18:39:11.553725 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.553692    2587 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:11.620727 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.620692    2587 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:11.637146 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.637120    2587 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:39:11.638090 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.638071    2587 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.ec2.internal"
Sep 17 18:39:11.652238 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.652207    2587 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:39:11.990366 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:11.990340    2587 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:12.096526 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.096469    2587 apiserver.go:52] "Watching apiserver"
Sep 17 18:39:12.104382 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.104351    2587 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 18:39:12.104728 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.104708    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-7frk4","openshift-network-operator/iptables-alerter-ms8k6","openshift-ovn-kubernetes/ovnkube-node-pzlnd","kube-system/global-pull-secret-syncer-9sj77","kube-system/konnectivity-agent-9nxm8","kube-system/kube-apiserver-proxy-ip-10-0-140-194.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb","openshift-cluster-node-tuning-operator/tuned-pdx8t","openshift-dns/node-resolver-nngbm","openshift-image-registry/node-ca-59srk","openshift-multus/multus-additional-cni-plugins-w4t6b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal","openshift-multus/multus-fp6c4","openshift-multus/network-metrics-daemon-7m9j8"]
Sep 17 18:39:12.107604 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.107583    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.110489 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.110140    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.110683 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.110663    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 18:39:12.110683 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.110676    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-889xn\""
Sep 17 18:39:12.110855 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.110663    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 18:39:12.111073 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.111055    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 18:39:12.112688 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.112263    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.112688 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.112274    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 18:39:12.112688 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.112296    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nqxpc\""
Sep 17 18:39:12.112688 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.112578    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 18:39:12.112930 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.112747    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 18:39:12.113207 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.113188    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 18:39:12.114545 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.114526    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:12.114652 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.114594    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:12.116548 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.116527    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 18:39:12.116903 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.116885    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9nxm8"
Sep 17 18:39:12.117833 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.117814    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qb2mk\""
Sep 17 18:39:12.118141 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.118123    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 18:39:12.118350 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.118332    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 18:39:12.118496 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.118483    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 18:39:12.118615 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.118588    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 18:39:12.118706 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.118695    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 18:39:12.119908 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.119409    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 18:39:12.119908 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.119590    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 18:39:12.119908 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.119737    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jj4dg\""
Sep 17 18:39:12.120550 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.120308    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:12.120550 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.120377    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:12.120731 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.120715    2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 18:39:12.125146 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.123613    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nngbm"
Sep 17 18:39:12.126215 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.126192    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 18:39:12.126390 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.126365    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5zppv\""
Sep 17 18:39:12.126557 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.126198    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:12.126669 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.126654    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 18:39:12.126773 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.126646    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:12.128998 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.128976    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ms8k6"
Sep 17 18:39:12.131392 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.131188    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:39:12.131392 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.131362    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 18:39:12.131655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.131632    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 18:39:12.131935 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.131918    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-59srk"
Sep 17 18:39:12.132028 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.132014    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6tsk8\""
Sep 17 18:39:12.132853 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.132834    2587 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:39:12.133298 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133102    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-run-ovn-kubernetes\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.133298 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133137    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75wwx\" (UniqueName: \"kubernetes.io/projected/9d5360ef-43f6-4b49-88bb-67b77b161636-kube-api-access-75wwx\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.133298 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133164    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ddd560fe-7729-4b3b-a307-d896e48b0baa-dbus\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:12.133298 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133187    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.133298 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133211    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d5360ef-43f6-4b49-88bb-67b77b161636-ovnkube-config\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.133298 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133260    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-system-cni-dir\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.133298 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133293    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-var-lib-cni-multus\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.133692 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133318    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-hostroot\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.133692 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133341    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-run-multus-certs\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.133692 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133372    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-registration-dir\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.133692 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133399    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-device-dir\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.133692 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133464    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-cni-bin\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.133692 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133502    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs\") pod \"network-metrics-daemon-7m9j8\" (UID: \"b425f704-e370-447b-a2f4-23d81057111d\") " pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:12.133692 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133529    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-multus-cni-dir\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.133692 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133552    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-multus-conf-dir\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.133692 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133578    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-etc-kubernetes\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.133692 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133602    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-sys-fs\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.133692 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133638    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-log-socket\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.133692 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133674    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-cni-netd\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133710    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-multus-daemon-config\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133734    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ddd560fe-7729-4b3b-a307-d896e48b0baa-kubelet-config\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133772    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f28c7ec2-e7d8-441d-ae45-781ddd349850-konnectivity-ca\") pod \"konnectivity-agent-9nxm8\" (UID: \"f28c7ec2-e7d8-441d-ae45-781ddd349850\") " pod="kube-system/konnectivity-agent-9nxm8"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133800    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-var-lib-openvswitch\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133827    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133845    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d5360ef-43f6-4b49-88bb-67b77b161636-ovnkube-script-lib\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133873    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-cnibin\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133897    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-var-lib-kubelet\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133919    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10bad8ad-782d-4a52-8450-c3399c47dc9b-hosts-file\") pod \"node-resolver-nngbm\" (UID: \"10bad8ad-782d-4a52-8450-c3399c47dc9b\") " pod="openshift-dns/node-resolver-nngbm"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133956    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-etc-openvswitch\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.133980    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-run-openvswitch\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134001    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d5360ef-43f6-4b49-88bb-67b77b161636-env-overrides\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134023    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d5360ef-43f6-4b49-88bb-67b77b161636-ovn-node-metrics-cert\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134046    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4lwx\" (UniqueName: \"kubernetes.io/projected/b425f704-e370-447b-a2f4-23d81057111d-kube-api-access-j4lwx\") pod \"network-metrics-daemon-7m9j8\" (UID: \"b425f704-e370-447b-a2f4-23d81057111d\") " pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134068    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-cni-binary-copy\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.134285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134096    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-run-k8s-cni-cncf-io\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134136    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-var-lib-cni-bin\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134162    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f28c7ec2-e7d8-441d-ae45-781ddd349850-agent-certs\") pod \"konnectivity-agent-9nxm8\" (UID: \"f28c7ec2-e7d8-441d-ae45-781ddd349850\") " pod="kube-system/konnectivity-agent-9nxm8"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134185    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5h6w\" (UniqueName: \"kubernetes.io/projected/10bad8ad-782d-4a52-8450-c3399c47dc9b-kube-api-access-c5h6w\") pod \"node-resolver-nngbm\" (UID: \"10bad8ad-782d-4a52-8450-c3399c47dc9b\") " pod="openshift-dns/node-resolver-nngbm"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134207    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-socket-dir\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134229    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-slash\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134251    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-run-netns\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134272    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-multus-socket-dir-parent\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134308    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-run-netns\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134329    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bad8ad-782d-4a52-8450-c3399c47dc9b-tmp-dir\") pod \"node-resolver-nngbm\" (UID: \"10bad8ad-782d-4a52-8450-c3399c47dc9b\") " pod="openshift-dns/node-resolver-nngbm"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134349    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134375    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rxmp\" (UniqueName: \"kubernetes.io/projected/87018d8d-b113-4d82-875a-48918709e1f4-kube-api-access-7rxmp\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134401    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-systemd-units\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134423    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-run-ovn\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134444    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-node-log\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134484    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk6rw\" (UniqueName: \"kubernetes.io/projected/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-kube-api-access-jk6rw\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.135026 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134518    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:12.135753 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134546    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-kubelet\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.135753 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.134569    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-run-systemd\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.135753 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.135151    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 18:39:12.135753 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.135532    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 18:39:12.135753 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.135678    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6j4b8\""
Sep 17 18:39:12.136001 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.135792    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 18:39:12.136001 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.135956    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.137732 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.137713    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.138584 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.138568    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-thh2h\""
Sep 17 18:39:12.138584 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.138577    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 18:39:12.138692 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.138620    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 18:39:12.140210 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.140191    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 18:39:12.140299 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.140228    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:39:12.140956 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.140789    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xhnps\""
Sep 17 18:39:12.165655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.165626    2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-44nc7"
Sep 17 18:39:12.174815 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.174792    2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-44nc7"
Sep 17 18:39:12.222450 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.222382    2587 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 18:39:12.235803 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.235771    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75b0ba53-5b75-493d-a60e-f479b53a0faa-cni-binary-copy\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.235985 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.235818    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/140238bf-b9c8-4b3f-87cc-158db60b6196-tmp\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.235985 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.235860    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-system-cni-dir\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.235985 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.235902    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-var-lib-cni-multus\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.235985 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.235925    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-device-dir\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.235985 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.235959    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs\") pod \"network-metrics-daemon-7m9j8\" (UID: \"b425f704-e370-447b-a2f4-23d81057111d\") " pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:12.235985 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.235982    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-run\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.236176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.235987    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-system-cni-dir\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.236176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.235996    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-sys\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.236176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.235993    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-var-lib-cni-multus\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.236176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236013    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmxr5\" (UniqueName: \"kubernetes.io/projected/aff06f71-dd37-479f-a73a-aaae682f8b78-kube-api-access-zmxr5\") pod \"iptables-alerter-ms8k6\" (UID: \"aff06f71-dd37-479f-a73a-aaae682f8b78\") " pod="openshift-network-operator/iptables-alerter-ms8k6"
Sep 17 18:39:12.236176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236032    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0010ab57-08d7-481d-aa7f-7c480d8ece9f-serviceca\") pod \"node-ca-59srk\" (UID: \"0010ab57-08d7-481d-aa7f-7c480d8ece9f\") " pod="openshift-image-registry/node-ca-59srk"
Sep 17 18:39:12.236176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236054    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-multus-conf-dir\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.236176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236066    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-device-dir\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.236176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236075    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-etc-kubernetes\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.236176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236096    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-log-socket\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.236176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236114    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-cni-netd\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.236176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236128    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-multus-conf-dir\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.236176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236136    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-systemd\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.236176 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.236158    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236195    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-log-socket\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236206    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-etc-kubernetes\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236160    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj4zg\" (UniqueName: \"kubernetes.io/projected/140238bf-b9c8-4b3f-87cc-158db60b6196-kube-api-access-zj4zg\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.236232    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs podName:b425f704-e370-447b-a2f4-23d81057111d nodeName:}" failed. No retries permitted until 2026-09-17 18:39:12.736209598 +0000 UTC m=+4.615178340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs") pod "network-metrics-daemon-7m9j8" (UID: "b425f704-e370-447b-a2f4-23d81057111d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236231    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-cni-netd\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236267    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aff06f71-dd37-479f-a73a-aaae682f8b78-host-slash\") pod \"iptables-alerter-ms8k6\" (UID: \"aff06f71-dd37-479f-a73a-aaae682f8b78\") " pod="openshift-network-operator/iptables-alerter-ms8k6"
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236297    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-multus-daemon-config\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236327    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f28c7ec2-e7d8-441d-ae45-781ddd349850-konnectivity-ca\") pod \"konnectivity-agent-9nxm8\" (UID: \"f28c7ec2-e7d8-441d-ae45-781ddd349850\") " pod="kube-system/konnectivity-agent-9nxm8"
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236400    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-var-lib-openvswitch\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236427    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236467    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d5360ef-43f6-4b49-88bb-67b77b161636-ovnkube-script-lib\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236489    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-etc-openvswitch\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236514    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-var-lib-openvswitch\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236529    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-run-openvswitch\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236557    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-run-k8s-cni-cncf-io\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.236632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236585    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-var-lib-cni-bin\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236608    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5h6w\" (UniqueName: \"kubernetes.io/projected/10bad8ad-782d-4a52-8450-c3399c47dc9b-kube-api-access-c5h6w\") pod \"node-resolver-nngbm\" (UID: \"10bad8ad-782d-4a52-8450-c3399c47dc9b\") " pod="openshift-dns/node-resolver-nngbm"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236618    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-etc-openvswitch\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236631    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-slash\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236669    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-run-netns\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236688    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-run-k8s-cni-cncf-io\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236694    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-run-ovn-kubernetes\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236727    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4lwx\" (UniqueName: \"kubernetes.io/projected/b425f704-e370-447b-a2f4-23d81057111d-kube-api-access-j4lwx\") pod \"network-metrics-daemon-7m9j8\" (UID: \"b425f704-e370-447b-a2f4-23d81057111d\") " pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236741    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-run-openvswitch\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236759    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-run-ovn\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236781    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-node-log\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236784    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-slash\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236806    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75b0ba53-5b75-493d-a60e-f479b53a0faa-system-cni-dir\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236829    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-var-lib-cni-bin\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236830    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75b0ba53-5b75-493d-a60e-f479b53a0faa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236890    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-run-ovn-kubernetes\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236894    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:12.237457 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236924    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-run-ovn\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236968    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75b0ba53-5b75-493d-a60e-f479b53a0faa-cnibin\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236998    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-sysctl-conf\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.236975    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237012    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f28c7ec2-e7d8-441d-ae45-781ddd349850-konnectivity-ca\") pod \"konnectivity-agent-9nxm8\" (UID: \"f28c7ec2-e7d8-441d-ae45-781ddd349850\") " pod="kube-system/konnectivity-agent-9nxm8"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237013    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-multus-daemon-config\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237017    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d5360ef-43f6-4b49-88bb-67b77b161636-ovnkube-script-lib\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.237066    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237100    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-host\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237146    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-run-netns\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237189    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-node-log\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237215    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aff06f71-dd37-479f-a73a-aaae682f8b78-iptables-alerter-script\") pod \"iptables-alerter-ms8k6\" (UID: \"aff06f71-dd37-479f-a73a-aaae682f8b78\") " pod="openshift-network-operator/iptables-alerter-ms8k6"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.237272    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret podName:ddd560fe-7729-4b3b-a307-d896e48b0baa nodeName:}" failed. No retries permitted until 2026-09-17 18:39:12.737238713 +0000 UTC m=+4.616207470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret") pod "global-pull-secret-syncer-9sj77" (UID: "ddd560fe-7729-4b3b-a307-d896e48b0baa") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237313    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237356    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-sysconfig\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237374    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-hostroot\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237390    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-run-multus-certs\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.238266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237412    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-registration-dir\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237435    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-cni-bin\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237442    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-hostroot\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237445    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237462    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-run-multus-certs\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237510    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-cni-bin\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237518    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-registration-dir\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237507    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75b0ba53-5b75-493d-a60e-f479b53a0faa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237567    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-multus-cni-dir\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237594    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-sys-fs\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237623    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-sysctl-d\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237653    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz8j4\" (UniqueName: \"kubernetes.io/projected/0010ab57-08d7-481d-aa7f-7c480d8ece9f-kube-api-access-rz8j4\") pod \"node-ca-59srk\" (UID: \"0010ab57-08d7-481d-aa7f-7c480d8ece9f\") " pod="openshift-image-registry/node-ca-59srk"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237679    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ddd560fe-7729-4b3b-a307-d896e48b0baa-kubelet-config\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237699    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-sys-fs\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237702    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-cnibin\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237735    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ddd560fe-7729-4b3b-a307-d896e48b0baa-kubelet-config\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237741    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-cnibin\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.239127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237744    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-var-lib-kubelet\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237749    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-multus-cni-dir\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237772    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10bad8ad-782d-4a52-8450-c3399c47dc9b-hosts-file\") pod \"node-resolver-nngbm\" (UID: \"10bad8ad-782d-4a52-8450-c3399c47dc9b\") " pod="openshift-dns/node-resolver-nngbm"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237802    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-var-lib-kubelet\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237807    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d5360ef-43f6-4b49-88bb-67b77b161636-env-overrides\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237855    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d5360ef-43f6-4b49-88bb-67b77b161636-ovn-node-metrics-cert\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237815    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10bad8ad-782d-4a52-8450-c3399c47dc9b-hosts-file\") pod \"node-resolver-nngbm\" (UID: \"10bad8ad-782d-4a52-8450-c3399c47dc9b\") " pod="openshift-dns/node-resolver-nngbm"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237882    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-kubernetes\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237924    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0010ab57-08d7-481d-aa7f-7c480d8ece9f-host\") pod \"node-ca-59srk\" (UID: \"0010ab57-08d7-481d-aa7f-7c480d8ece9f\") " pod="openshift-image-registry/node-ca-59srk"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.237972    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-cni-binary-copy\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238001    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f28c7ec2-e7d8-441d-ae45-781ddd349850-agent-certs\") pod \"konnectivity-agent-9nxm8\" (UID: \"f28c7ec2-e7d8-441d-ae45-781ddd349850\") " pod="kube-system/konnectivity-agent-9nxm8"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238030    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-socket-dir\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238055    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75b0ba53-5b75-493d-a60e-f479b53a0faa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238084    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-lib-modules\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238108    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-multus-socket-dir-parent\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238132    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-run-netns\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238144    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d5360ef-43f6-4b49-88bb-67b77b161636-env-overrides\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.239893 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238160    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bad8ad-782d-4a52-8450-c3399c47dc9b-tmp-dir\") pod \"node-resolver-nngbm\" (UID: \"10bad8ad-782d-4a52-8450-c3399c47dc9b\") " pod="openshift-dns/node-resolver-nngbm"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238186    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238214    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rxmp\" (UniqueName: \"kubernetes.io/projected/87018d8d-b113-4d82-875a-48918709e1f4-kube-api-access-7rxmp\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238237    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-systemd-units\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238248    2587 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238265    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9h2p\" (UniqueName: \"kubernetes.io/projected/75b0ba53-5b75-493d-a60e-f479b53a0faa-kube-api-access-n9h2p\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238317    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-var-lib-kubelet\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238343    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk6rw\" (UniqueName: \"kubernetes.io/projected/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-kube-api-access-jk6rw\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238367    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-kubelet\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238394    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-run-systemd\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238424    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75wwx\" (UniqueName: \"kubernetes.io/projected/9d5360ef-43f6-4b49-88bb-67b77b161636-kube-api-access-75wwx\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238452    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qws\" (UniqueName: \"kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws\") pod \"network-check-target-7frk4\" (UID: \"3821fdb9-ccfa-42c2-b476-9c624935e87e\") " pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238504    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-modprobe-d\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238531    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-tuned\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238553    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238559    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ddd560fe-7729-4b3b-a307-d896e48b0baa-dbus\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238588    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-cni-binary-copy\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.240745 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238650    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-multus-socket-dir-parent\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.241546 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238654    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ddd560fe-7729-4b3b-a307-d896e48b0baa-dbus\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:12.241546 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238696    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-host-run-netns\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.241546 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238700    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10bad8ad-782d-4a52-8450-c3399c47dc9b-tmp-dir\") pod \"node-resolver-nngbm\" (UID: \"10bad8ad-782d-4a52-8450-c3399c47dc9b\") " pod="openshift-dns/node-resolver-nngbm"
Sep 17 18:39:12.241546 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238692    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/87018d8d-b113-4d82-875a-48918709e1f4-socket-dir\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.241546 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238779    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d5360ef-43f6-4b49-88bb-67b77b161636-ovnkube-config\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.241546 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238821    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-run-systemd\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.241546 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.238900    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-host-kubelet\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.241546 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.239152    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d5360ef-43f6-4b49-88bb-67b77b161636-systemd-units\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.241546 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.239463    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d5360ef-43f6-4b49-88bb-67b77b161636-ovnkube-config\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.242576 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.242556    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d5360ef-43f6-4b49-88bb-67b77b161636-ovn-node-metrics-cert\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.242685 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.242652    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f28c7ec2-e7d8-441d-ae45-781ddd349850-agent-certs\") pod \"konnectivity-agent-9nxm8\" (UID: \"f28c7ec2-e7d8-441d-ae45-781ddd349850\") " pod="kube-system/konnectivity-agent-9nxm8"
Sep 17 18:39:12.252241 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.249160    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4lwx\" (UniqueName: \"kubernetes.io/projected/b425f704-e370-447b-a2f4-23d81057111d-kube-api-access-j4lwx\") pod \"network-metrics-daemon-7m9j8\" (UID: \"b425f704-e370-447b-a2f4-23d81057111d\") " pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:12.252241 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.249597    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk6rw\" (UniqueName: \"kubernetes.io/projected/05ce87e8-6de7-4fd2-bf76-ecef629fc3b2-kube-api-access-jk6rw\") pod \"multus-fp6c4\" (UID: \"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2\") " pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.252241 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.251538    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rxmp\" (UniqueName: \"kubernetes.io/projected/87018d8d-b113-4d82-875a-48918709e1f4-kube-api-access-7rxmp\") pod \"aws-ebs-csi-driver-node-f7jsb\" (UID: \"87018d8d-b113-4d82-875a-48918709e1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.252241 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.251834    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5h6w\" (UniqueName: \"kubernetes.io/projected/10bad8ad-782d-4a52-8450-c3399c47dc9b-kube-api-access-c5h6w\") pod \"node-resolver-nngbm\" (UID: \"10bad8ad-782d-4a52-8450-c3399c47dc9b\") " pod="openshift-dns/node-resolver-nngbm"
Sep 17 18:39:12.254970 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.254244    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75wwx\" (UniqueName: \"kubernetes.io/projected/9d5360ef-43f6-4b49-88bb-67b77b161636-kube-api-access-75wwx\") pod \"ovnkube-node-pzlnd\" (UID: \"9d5360ef-43f6-4b49-88bb-67b77b161636\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.257417 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.257367    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.ec2.internal" event={"ID":"fdc7ceb412f9c95a22d608979ff76c2f","Type":"ContainerStarted","Data":"fb9bf6e7c30be2e4855d8959860027211d6820c62151c56cc7897597962651fa"}
Sep 17 18:39:12.258696 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.258675    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal" event={"ID":"9ce1f0d6f55636400e257394b4e53616","Type":"ContainerStarted","Data":"b067b87c70e9df2f0b1a9c7f355aa696fdb5a899f3c56255adb26c55246c1893"}
Sep 17 18:39:12.339379 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339334    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75b0ba53-5b75-493d-a60e-f479b53a0faa-system-cni-dir\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.339379 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339380    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75b0ba53-5b75-493d-a60e-f479b53a0faa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.339608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339424    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75b0ba53-5b75-493d-a60e-f479b53a0faa-cnibin\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.339608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339447    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-sysctl-conf\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.339608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339452    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75b0ba53-5b75-493d-a60e-f479b53a0faa-system-cni-dir\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.339608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339473    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-host\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.339608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339498    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aff06f71-dd37-479f-a73a-aaae682f8b78-iptables-alerter-script\") pod \"iptables-alerter-ms8k6\" (UID: \"aff06f71-dd37-479f-a73a-aaae682f8b78\") " pod="openshift-network-operator/iptables-alerter-ms8k6"
Sep 17 18:39:12.339608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339512    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75b0ba53-5b75-493d-a60e-f479b53a0faa-cnibin\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.339608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339522    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-sysconfig\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.339608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339563    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-host\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.339608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339581    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-sysconfig\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.339608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339585    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75b0ba53-5b75-493d-a60e-f479b53a0faa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339624    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-sysctl-d\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339644    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-sysctl-conf\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339649    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz8j4\" (UniqueName: \"kubernetes.io/projected/0010ab57-08d7-481d-aa7f-7c480d8ece9f-kube-api-access-rz8j4\") pod \"node-ca-59srk\" (UID: \"0010ab57-08d7-481d-aa7f-7c480d8ece9f\") " pod="openshift-image-registry/node-ca-59srk"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339785    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-kubernetes\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339813    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0010ab57-08d7-481d-aa7f-7c480d8ece9f-host\") pod \"node-ca-59srk\" (UID: \"0010ab57-08d7-481d-aa7f-7c480d8ece9f\") " pod="openshift-image-registry/node-ca-59srk"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339838    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-sysctl-d\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339853    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75b0ba53-5b75-493d-a60e-f479b53a0faa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339878    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-lib-modules\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339891    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-kubernetes\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339895    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0010ab57-08d7-481d-aa7f-7c480d8ece9f-host\") pod \"node-ca-59srk\" (UID: \"0010ab57-08d7-481d-aa7f-7c480d8ece9f\") " pod="openshift-image-registry/node-ca-59srk"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340001    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-lib-modules\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.339899    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9h2p\" (UniqueName: \"kubernetes.io/projected/75b0ba53-5b75-493d-a60e-f479b53a0faa-kube-api-access-n9h2p\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340039    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-var-lib-kubelet\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340061    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75b0ba53-5b75-493d-a60e-f479b53a0faa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340068    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qws\" (UniqueName: \"kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws\") pod \"network-check-target-7frk4\" (UID: \"3821fdb9-ccfa-42c2-b476-9c624935e87e\") " pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:12.340109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340100    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-modprobe-d\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340127    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-tuned\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340150    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75b0ba53-5b75-493d-a60e-f479b53a0faa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340154    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75b0ba53-5b75-493d-a60e-f479b53a0faa-cni-binary-copy\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340153    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75b0ba53-5b75-493d-a60e-f479b53a0faa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340185    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aff06f71-dd37-479f-a73a-aaae682f8b78-iptables-alerter-script\") pod \"iptables-alerter-ms8k6\" (UID: \"aff06f71-dd37-479f-a73a-aaae682f8b78\") " pod="openshift-network-operator/iptables-alerter-ms8k6"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340225    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-modprobe-d\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340225    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-var-lib-kubelet\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340259    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/140238bf-b9c8-4b3f-87cc-158db60b6196-tmp\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340287    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-run\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340308    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-sys\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340324    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmxr5\" (UniqueName: \"kubernetes.io/projected/aff06f71-dd37-479f-a73a-aaae682f8b78-kube-api-access-zmxr5\") pod \"iptables-alerter-ms8k6\" (UID: \"aff06f71-dd37-479f-a73a-aaae682f8b78\") " pod="openshift-network-operator/iptables-alerter-ms8k6"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340339    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0010ab57-08d7-481d-aa7f-7c480d8ece9f-serviceca\") pod \"node-ca-59srk\" (UID: \"0010ab57-08d7-481d-aa7f-7c480d8ece9f\") " pod="openshift-image-registry/node-ca-59srk"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340397    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-sys\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340415    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-run\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340562    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-systemd\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340599    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj4zg\" (UniqueName: \"kubernetes.io/projected/140238bf-b9c8-4b3f-87cc-158db60b6196-kube-api-access-zj4zg\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.340839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340604    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75b0ba53-5b75-493d-a60e-f479b53a0faa-cni-binary-copy\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.341403 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340624    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aff06f71-dd37-479f-a73a-aaae682f8b78-host-slash\") pod \"iptables-alerter-ms8k6\" (UID: \"aff06f71-dd37-479f-a73a-aaae682f8b78\") " pod="openshift-network-operator/iptables-alerter-ms8k6"
Sep 17 18:39:12.341403 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340661    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-systemd\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.341403 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340710    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aff06f71-dd37-479f-a73a-aaae682f8b78-host-slash\") pod \"iptables-alerter-ms8k6\" (UID: \"aff06f71-dd37-479f-a73a-aaae682f8b78\") " pod="openshift-network-operator/iptables-alerter-ms8k6"
Sep 17 18:39:12.341403 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.340880    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0010ab57-08d7-481d-aa7f-7c480d8ece9f-serviceca\") pod \"node-ca-59srk\" (UID: \"0010ab57-08d7-481d-aa7f-7c480d8ece9f\") " pod="openshift-image-registry/node-ca-59srk"
Sep 17 18:39:12.342635 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.342610    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/140238bf-b9c8-4b3f-87cc-158db60b6196-tmp\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.342735 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.342649    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/140238bf-b9c8-4b3f-87cc-158db60b6196-etc-tuned\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.351722 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.351697    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9h2p\" (UniqueName: \"kubernetes.io/projected/75b0ba53-5b75-493d-a60e-f479b53a0faa-kube-api-access-n9h2p\") pod \"multus-additional-cni-plugins-w4t6b\" (UID: \"75b0ba53-5b75-493d-a60e-f479b53a0faa\") " pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.352218 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.352196    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:12.352218 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.352219    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:12.352380 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.352228    2587 projected.go:196] Error preparing data for projected volume kube-api-access-d5qws for pod openshift-network-diagnostics/network-check-target-7frk4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:12.352380 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.352304    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws podName:3821fdb9-ccfa-42c2-b476-9c624935e87e nodeName:}" failed. No retries permitted until 2026-09-17 18:39:12.852289866 +0000 UTC m=+4.731258601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d5qws" (UniqueName: "kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws") pod "network-check-target-7frk4" (UID: "3821fdb9-ccfa-42c2-b476-9c624935e87e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:12.352608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.352588    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmxr5\" (UniqueName: \"kubernetes.io/projected/aff06f71-dd37-479f-a73a-aaae682f8b78-kube-api-access-zmxr5\") pod \"iptables-alerter-ms8k6\" (UID: \"aff06f71-dd37-479f-a73a-aaae682f8b78\") " pod="openshift-network-operator/iptables-alerter-ms8k6"
Sep 17 18:39:12.353802 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.353781    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz8j4\" (UniqueName: \"kubernetes.io/projected/0010ab57-08d7-481d-aa7f-7c480d8ece9f-kube-api-access-rz8j4\") pod \"node-ca-59srk\" (UID: \"0010ab57-08d7-481d-aa7f-7c480d8ece9f\") " pod="openshift-image-registry/node-ca-59srk"
Sep 17 18:39:12.353903 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.353817    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj4zg\" (UniqueName: \"kubernetes.io/projected/140238bf-b9c8-4b3f-87cc-158db60b6196-kube-api-access-zj4zg\") pod \"tuned-pdx8t\" (UID: \"140238bf-b9c8-4b3f-87cc-158db60b6196\") " pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.419840 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.419813    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb"
Sep 17 18:39:12.427377 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.427347    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fp6c4"
Sep 17 18:39:12.438249 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.438211    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:12.446092 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.446068    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9nxm8"
Sep 17 18:39:12.452860 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.452841    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nngbm"
Sep 17 18:39:12.460526 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.460499    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ms8k6"
Sep 17 18:39:12.468102 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.468073    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-59srk"
Sep 17 18:39:12.474776 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.474725    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w4t6b"
Sep 17 18:39:12.481437 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.481418    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pdx8t"
Sep 17 18:39:12.743082 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.742992    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:12.743214 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.743088    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs\") pod \"network-metrics-daemon-7m9j8\" (UID: \"b425f704-e370-447b-a2f4-23d81057111d\") " pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:12.743214 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.743178    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:12.743214 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.743199    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:12.743354 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.743257    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret podName:ddd560fe-7729-4b3b-a307-d896e48b0baa nodeName:}" failed. No retries permitted until 2026-09-17 18:39:13.74323743 +0000 UTC m=+5.622206186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret") pod "global-pull-secret-syncer-9sj77" (UID: "ddd560fe-7729-4b3b-a307-d896e48b0baa") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:12.743354 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.743276    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs podName:b425f704-e370-447b-a2f4-23d81057111d nodeName:}" failed. No retries permitted until 2026-09-17 18:39:13.743267508 +0000 UTC m=+5.622236278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs") pod "network-metrics-daemon-7m9j8" (UID: "b425f704-e370-447b-a2f4-23d81057111d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:12.944810 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:12.944774    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qws\" (UniqueName: \"kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws\") pod \"network-check-target-7frk4\" (UID: \"3821fdb9-ccfa-42c2-b476-9c624935e87e\") " pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:12.944985 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.944929    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:12.944985 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.944966    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:12.944985 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.944975    2587 projected.go:196] Error preparing data for projected volume kube-api-access-d5qws for pod openshift-network-diagnostics/network-check-target-7frk4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:12.945107 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:12.945038    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws podName:3821fdb9-ccfa-42c2-b476-9c624935e87e nodeName:}" failed. No retries permitted until 2026-09-17 18:39:13.945020654 +0000 UTC m=+5.823989388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5qws" (UniqueName: "kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws") pod "network-check-target-7frk4" (UID: "3821fdb9-ccfa-42c2-b476-9c624935e87e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:13.178117 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.178019    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:34:12 +0000 UTC" deadline="2028-02-23 00:19:29.823693689 +0000 UTC"
Sep 17 18:39:13.178117 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.178072    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12557h40m16.645624577s"
Sep 17 18:39:13.222477 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:13.222446    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b0ba53_5b75_493d_a60e_f479b53a0faa.slice/crio-b9a1dd1cc9aab17079bb5e1240ff2476ab21647619f93247f2ac7d21fe2abc51 WatchSource:0}: Error finding container b9a1dd1cc9aab17079bb5e1240ff2476ab21647619f93247f2ac7d21fe2abc51: Status 404 returned error can't find the container with id b9a1dd1cc9aab17079bb5e1240ff2476ab21647619f93247f2ac7d21fe2abc51
Sep 17 18:39:13.223188 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:13.223166    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140238bf_b9c8_4b3f_87cc_158db60b6196.slice/crio-8da7cc7841fea2e7a3ca6ea0560d093e1d40dd145aa05133be8d66a1f460d736 WatchSource:0}: Error finding container 8da7cc7841fea2e7a3ca6ea0560d093e1d40dd145aa05133be8d66a1f460d736: Status 404 returned error can't find the container with id 8da7cc7841fea2e7a3ca6ea0560d093e1d40dd145aa05133be8d66a1f460d736
Sep 17 18:39:13.223995 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:13.223933    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87018d8d_b113_4d82_875a_48918709e1f4.slice/crio-71af762707faa7aebabe6c722386adde3369aa807f16dc2b3e7b5995b5bb6f5f WatchSource:0}: Error finding container 71af762707faa7aebabe6c722386adde3369aa807f16dc2b3e7b5995b5bb6f5f: Status 404 returned error can't find the container with id 71af762707faa7aebabe6c722386adde3369aa807f16dc2b3e7b5995b5bb6f5f
Sep 17 18:39:13.224834 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:13.224810    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10bad8ad_782d_4a52_8450_c3399c47dc9b.slice/crio-7b5f1eef52c2973e4e40db5fcf65cc5d7185d481bdfafbc700434c98e68ecfeb WatchSource:0}: Error finding container 7b5f1eef52c2973e4e40db5fcf65cc5d7185d481bdfafbc700434c98e68ecfeb: Status 404 returned error can't find the container with id 7b5f1eef52c2973e4e40db5fcf65cc5d7185d481bdfafbc700434c98e68ecfeb
Sep 17 18:39:13.226768 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:13.226652    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5360ef_43f6_4b49_88bb_67b77b161636.slice/crio-6863138817a63ec6bc0cd032ffd9e2f9e21f065c85784bfae5bfdb54a585676d WatchSource:0}: Error finding container 6863138817a63ec6bc0cd032ffd9e2f9e21f065c85784bfae5bfdb54a585676d: Status 404 returned error can't find the container with id 6863138817a63ec6bc0cd032ffd9e2f9e21f065c85784bfae5bfdb54a585676d
Sep 17 18:39:13.229491 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:13.227666    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28c7ec2_e7d8_441d_ae45_781ddd349850.slice/crio-5aa5143b0e63449ada7f4e23c12b0fc7490dec518dfd9ef3c2890c4d331911d8 WatchSource:0}: Error finding container 5aa5143b0e63449ada7f4e23c12b0fc7490dec518dfd9ef3c2890c4d331911d8: Status 404 returned error can't find the container with id 5aa5143b0e63449ada7f4e23c12b0fc7490dec518dfd9ef3c2890c4d331911d8
Sep 17 18:39:13.230291 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:13.230269    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff06f71_dd37_479f_a73a_aaae682f8b78.slice/crio-b08e0559ef6636890e0351c2d1fcfa93a66d39b4b1e0158b0ed14ade9bab40b2 WatchSource:0}: Error finding container b08e0559ef6636890e0351c2d1fcfa93a66d39b4b1e0158b0ed14ade9bab40b2: Status 404 returned error can't find the container with id b08e0559ef6636890e0351c2d1fcfa93a66d39b4b1e0158b0ed14ade9bab40b2
Sep 17 18:39:13.232471 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:13.232424    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0010ab57_08d7_481d_aa7f_7c480d8ece9f.slice/crio-3987c1214e423b13d7c232a5c705c22832bbcb28e9519bde5237d4a61628bb86 WatchSource:0}: Error finding container 3987c1214e423b13d7c232a5c705c22832bbcb28e9519bde5237d4a61628bb86: Status 404 returned error can't find the container with id 3987c1214e423b13d7c232a5c705c22832bbcb28e9519bde5237d4a61628bb86
Sep 17 18:39:13.233014 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:13.232993    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ce87e8_6de7_4fd2_bf76_ecef629fc3b2.slice/crio-a7e0049e6e5f180a4db22241d8e86efbdc96d603dd05dd9bff9e5797aa0bea33 WatchSource:0}: Error finding container a7e0049e6e5f180a4db22241d8e86efbdc96d603dd05dd9bff9e5797aa0bea33: Status 404 returned error can't find the container with id a7e0049e6e5f180a4db22241d8e86efbdc96d603dd05dd9bff9e5797aa0bea33
Sep 17 18:39:13.264886 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.264848    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd" event={"ID":"9d5360ef-43f6-4b49-88bb-67b77b161636","Type":"ContainerStarted","Data":"6863138817a63ec6bc0cd032ffd9e2f9e21f065c85784bfae5bfdb54a585676d"}
Sep 17 18:39:13.265992 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.265969    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb" event={"ID":"87018d8d-b113-4d82-875a-48918709e1f4","Type":"ContainerStarted","Data":"71af762707faa7aebabe6c722386adde3369aa807f16dc2b3e7b5995b5bb6f5f"}
Sep 17 18:39:13.267271 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.267239    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pdx8t" event={"ID":"140238bf-b9c8-4b3f-87cc-158db60b6196","Type":"ContainerStarted","Data":"8da7cc7841fea2e7a3ca6ea0560d093e1d40dd145aa05133be8d66a1f460d736"}
Sep 17 18:39:13.268310 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.268288    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4t6b" event={"ID":"75b0ba53-5b75-493d-a60e-f479b53a0faa","Type":"ContainerStarted","Data":"b9a1dd1cc9aab17079bb5e1240ff2476ab21647619f93247f2ac7d21fe2abc51"}
Sep 17 18:39:13.269415 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.269390    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-59srk" event={"ID":"0010ab57-08d7-481d-aa7f-7c480d8ece9f","Type":"ContainerStarted","Data":"3987c1214e423b13d7c232a5c705c22832bbcb28e9519bde5237d4a61628bb86"}
Sep 17 18:39:13.271094 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.271071    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9nxm8" event={"ID":"f28c7ec2-e7d8-441d-ae45-781ddd349850","Type":"ContainerStarted","Data":"5aa5143b0e63449ada7f4e23c12b0fc7490dec518dfd9ef3c2890c4d331911d8"}
Sep 17 18:39:13.275062 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.275033    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nngbm" event={"ID":"10bad8ad-782d-4a52-8450-c3399c47dc9b","Type":"ContainerStarted","Data":"7b5f1eef52c2973e4e40db5fcf65cc5d7185d481bdfafbc700434c98e68ecfeb"}
Sep 17 18:39:13.278143 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.278125    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fp6c4" event={"ID":"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2","Type":"ContainerStarted","Data":"a7e0049e6e5f180a4db22241d8e86efbdc96d603dd05dd9bff9e5797aa0bea33"}
Sep 17 18:39:13.279332 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.279309    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ms8k6" event={"ID":"aff06f71-dd37-479f-a73a-aaae682f8b78","Type":"ContainerStarted","Data":"b08e0559ef6636890e0351c2d1fcfa93a66d39b4b1e0158b0ed14ade9bab40b2"}
Sep 17 18:39:13.752168 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.751852    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs\") pod \"network-metrics-daemon-7m9j8\" (UID: \"b425f704-e370-447b-a2f4-23d81057111d\") " pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:13.752168 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:13.752150    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:13.752381 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.752174    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:13.752381 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:13.752228    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs podName:b425f704-e370-447b-a2f4-23d81057111d nodeName:}" failed. No retries permitted until 2026-09-17 18:39:15.752208379 +0000 UTC m=+7.631177117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs") pod "network-metrics-daemon-7m9j8" (UID: "b425f704-e370-447b-a2f4-23d81057111d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:13.752381 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:13.752303    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:13.752381 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:13.752357    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret podName:ddd560fe-7729-4b3b-a307-d896e48b0baa nodeName:}" failed. No retries permitted until 2026-09-17 18:39:15.752341673 +0000 UTC m=+7.631310425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret") pod "global-pull-secret-syncer-9sj77" (UID: "ddd560fe-7729-4b3b-a307-d896e48b0baa") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:13.821301 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.821264    2587 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:39:13.953621 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:13.953583    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qws\" (UniqueName: \"kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws\") pod \"network-check-target-7frk4\" (UID: \"3821fdb9-ccfa-42c2-b476-9c624935e87e\") " pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:13.953837 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:13.953799    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:13.953837 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:13.953820    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:13.953837 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:13.953833    2587 projected.go:196] Error preparing data for projected volume kube-api-access-d5qws for pod openshift-network-diagnostics/network-check-target-7frk4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:13.954107 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:13.953894    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws podName:3821fdb9-ccfa-42c2-b476-9c624935e87e nodeName:}" failed. No retries permitted until 2026-09-17 18:39:15.953875222 +0000 UTC m=+7.832843974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5qws" (UniqueName: "kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws") pod "network-check-target-7frk4" (UID: "3821fdb9-ccfa-42c2-b476-9c624935e87e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:14.178567 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:14.178425    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:34:12 +0000 UTC" deadline="2028-04-24 03:44:15.8082201 +0000 UTC"
Sep 17 18:39:14.178567 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:14.178469    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14025h5m1.629756353s"
Sep 17 18:39:14.252613 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:14.252574    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:14.252791 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:14.252712    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:14.253221 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:14.253192    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:14.253320 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:14.253294    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:14.253480 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:14.253439    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:14.253557 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:14.253531    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:14.302988 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:14.302414    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.ec2.internal" event={"ID":"fdc7ceb412f9c95a22d608979ff76c2f","Type":"ContainerStarted","Data":"a64ac016e59c7a046cb821c74c0081603982dcf66346309aa706b2874fba1eff"}
Sep 17 18:39:14.316326 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:14.316291    2587 generic.go:356] "Generic (PLEG): container finished" podID="9ce1f0d6f55636400e257394b4e53616" containerID="57cac3131401527923e0ba72f5d61f4c46f8d9e03c16be9f77d640cdb2b061f7" exitCode=0
Sep 17 18:39:14.316478 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:14.316341    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal" event={"ID":"9ce1f0d6f55636400e257394b4e53616","Type":"ContainerDied","Data":"57cac3131401527923e0ba72f5d61f4c46f8d9e03c16be9f77d640cdb2b061f7"}
Sep 17 18:39:14.344458 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:14.343291    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-194.ec2.internal" podStartSLOduration=3.343270457 podStartE2EDuration="3.343270457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:14.322164513 +0000 UTC m=+6.201133270" watchObservedRunningTime="2026-09-17 18:39:14.343270457 +0000 UTC m=+6.222239214"
Sep 17 18:39:15.326430 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:15.326318    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal" event={"ID":"9ce1f0d6f55636400e257394b4e53616","Type":"ContainerStarted","Data":"5da2c3cc97b4273e45f05ff5156fbff91e2ff7f5c08f59e8cb28acb789b70094"}
Sep 17 18:39:15.768968 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:15.768619    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs\") pod \"network-metrics-daemon-7m9j8\" (UID: \"b425f704-e370-447b-a2f4-23d81057111d\") " pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:15.768968 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:15.768684    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:15.768968 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:15.768789    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:15.768968 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:15.768790    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:15.768968 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:15.768850    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret podName:ddd560fe-7729-4b3b-a307-d896e48b0baa nodeName:}" failed. No retries permitted until 2026-09-17 18:39:19.768832663 +0000 UTC m=+11.647801414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret") pod "global-pull-secret-syncer-9sj77" (UID: "ddd560fe-7729-4b3b-a307-d896e48b0baa") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:15.768968 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:15.768867    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs podName:b425f704-e370-447b-a2f4-23d81057111d nodeName:}" failed. No retries permitted until 2026-09-17 18:39:19.768858227 +0000 UTC m=+11.647826967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs") pod "network-metrics-daemon-7m9j8" (UID: "b425f704-e370-447b-a2f4-23d81057111d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:15.970307 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:15.970273    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qws\" (UniqueName: \"kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws\") pod \"network-check-target-7frk4\" (UID: \"3821fdb9-ccfa-42c2-b476-9c624935e87e\") " pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:15.970484 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:15.970403    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:15.970484 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:15.970417    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:15.970484 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:15.970426    2587 projected.go:196] Error preparing data for projected volume kube-api-access-d5qws for pod openshift-network-diagnostics/network-check-target-7frk4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:15.970484 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:15.970472    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws podName:3821fdb9-ccfa-42c2-b476-9c624935e87e nodeName:}" failed. No retries permitted until 2026-09-17 18:39:19.970458848 +0000 UTC m=+11.849427583 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5qws" (UniqueName: "kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws") pod "network-check-target-7frk4" (UID: "3821fdb9-ccfa-42c2-b476-9c624935e87e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:16.253161 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:16.253078    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:16.253333 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:16.253226    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:16.253632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:16.253611    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:16.253771 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:16.253703    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:16.255930 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:16.255907    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:16.256054 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:16.256032    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:18.255512 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:18.255481    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:18.255991 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:18.255619    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:18.256079 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:18.255996    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:18.256141 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:18.256088    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:18.256141 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:18.256133    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:18.256244 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:18.256200    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:19.808229 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:19.807320    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs\") pod \"network-metrics-daemon-7m9j8\" (UID: \"b425f704-e370-447b-a2f4-23d81057111d\") " pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:19.808229 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:19.807417    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:19.808229 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:19.807553    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:19.808229 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:19.807615    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret podName:ddd560fe-7729-4b3b-a307-d896e48b0baa nodeName:}" failed. No retries permitted until 2026-09-17 18:39:27.807596604 +0000 UTC m=+19.686565346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret") pod "global-pull-secret-syncer-9sj77" (UID: "ddd560fe-7729-4b3b-a307-d896e48b0baa") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:19.808229 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:19.808135    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:19.808229 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:19.808192    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs podName:b425f704-e370-447b-a2f4-23d81057111d nodeName:}" failed. No retries permitted until 2026-09-17 18:39:27.808175205 +0000 UTC m=+19.687143963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs") pod "network-metrics-daemon-7m9j8" (UID: "b425f704-e370-447b-a2f4-23d81057111d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:20.009131 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:20.009091    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qws\" (UniqueName: \"kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws\") pod \"network-check-target-7frk4\" (UID: \"3821fdb9-ccfa-42c2-b476-9c624935e87e\") " pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:20.009300 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:20.009248    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:20.009300 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:20.009269    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:20.009300 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:20.009279    2587 projected.go:196] Error preparing data for projected volume kube-api-access-d5qws for pod openshift-network-diagnostics/network-check-target-7frk4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:20.009469 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:20.009326    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws podName:3821fdb9-ccfa-42c2-b476-9c624935e87e nodeName:}" failed. No retries permitted until 2026-09-17 18:39:28.009311402 +0000 UTC m=+19.888280142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5qws" (UniqueName: "kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws") pod "network-check-target-7frk4" (UID: "3821fdb9-ccfa-42c2-b476-9c624935e87e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:20.254419 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:20.253569    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:20.254419 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:20.253705    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:20.254419 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:20.254093    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:20.254419 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:20.254183    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:20.254419 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:20.254256    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:20.254419 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:20.254338    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:22.252998 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:22.252952    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:22.256509 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:22.253265    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:22.256509 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:22.253558    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:22.256509 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:22.253648    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:22.256509 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:22.252958    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:22.256509 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:22.253794    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:24.252739 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:24.252411    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:24.252739 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:24.252446    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:24.252739 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:24.252554    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:24.252739 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:24.252601    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:24.252739 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:24.252624    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:24.252739 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:24.252713    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:26.252191 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:26.252151    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:26.252643 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:26.252152    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:26.252643 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:26.252152    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:26.252643 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:26.252385    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:26.252643 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:26.252477    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:26.252859 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:26.252297    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:27.864335 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:27.864294    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs\") pod \"network-metrics-daemon-7m9j8\" (UID: \"b425f704-e370-447b-a2f4-23d81057111d\") " pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:27.864787 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:27.864361    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:27.864787 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:27.864461    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:27.864787 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:27.864473    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:27.864787 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:27.864542    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs podName:b425f704-e370-447b-a2f4-23d81057111d nodeName:}" failed. No retries permitted until 2026-09-17 18:39:43.86452379 +0000 UTC m=+35.743492530 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs") pod "network-metrics-daemon-7m9j8" (UID: "b425f704-e370-447b-a2f4-23d81057111d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:27.864787 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:27.864559    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret podName:ddd560fe-7729-4b3b-a307-d896e48b0baa nodeName:}" failed. No retries permitted until 2026-09-17 18:39:43.864551805 +0000 UTC m=+35.743520544 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret") pod "global-pull-secret-syncer-9sj77" (UID: "ddd560fe-7729-4b3b-a307-d896e48b0baa") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:28.066245 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:28.066206    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qws\" (UniqueName: \"kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws\") pod \"network-check-target-7frk4\" (UID: \"3821fdb9-ccfa-42c2-b476-9c624935e87e\") " pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:28.066436 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:28.066417    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:28.066511 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:28.066447    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:28.066511 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:28.066460    2587 projected.go:196] Error preparing data for projected volume kube-api-access-d5qws for pod openshift-network-diagnostics/network-check-target-7frk4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:28.066645 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:28.066530    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws podName:3821fdb9-ccfa-42c2-b476-9c624935e87e nodeName:}" failed. No retries permitted until 2026-09-17 18:39:44.066510951 +0000 UTC m=+35.945479706 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5qws" (UniqueName: "kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws") pod "network-check-target-7frk4" (UID: "3821fdb9-ccfa-42c2-b476-9c624935e87e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:28.252556 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:28.252479    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:28.252556 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:28.252521    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:28.252776 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:28.252477    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:28.252776 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:28.252617    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:28.252776 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:28.252691    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:28.252776 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:28.252762    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:30.254134 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:30.254099    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:30.254549 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:30.254213    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:30.254549 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:30.254227    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:30.254549 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:30.254229    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:30.254549 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:30.254299    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:30.254549 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:30.254401    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:31.365384 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.365127    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pdx8t" event={"ID":"140238bf-b9c8-4b3f-87cc-158db60b6196","Type":"ContainerStarted","Data":"a366a9eb3dfd25fb27c03182bddb6e8fb98633fb0d144f49c72e2a06b35025b3"}
Sep 17 18:39:31.367498 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.367474    2587 generic.go:356] "Generic (PLEG): container finished" podID="75b0ba53-5b75-493d-a60e-f479b53a0faa" containerID="10c594859b57e74e8aff5bdaaa28dff5a1666ff611bf736402e0f735171b28e6" exitCode=0
Sep 17 18:39:31.367611 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.367567    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4t6b" event={"ID":"75b0ba53-5b75-493d-a60e-f479b53a0faa","Type":"ContainerDied","Data":"10c594859b57e74e8aff5bdaaa28dff5a1666ff611bf736402e0f735171b28e6"}
Sep 17 18:39:31.369658 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.369352    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-59srk" event={"ID":"0010ab57-08d7-481d-aa7f-7c480d8ece9f","Type":"ContainerStarted","Data":"7bb32db62f06c10f74e1ac1be04158e6a007c7d1ad62ebfa3f08e647e9dd744f"}
Sep 17 18:39:31.371277 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.371252    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9nxm8" event={"ID":"f28c7ec2-e7d8-441d-ae45-781ddd349850","Type":"ContainerStarted","Data":"f036c8355b9753fe2a5ec83a11ef8b16034a70202b410a449f4e87a855f582ab"}
Sep 17 18:39:31.372850 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.372830    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nngbm" event={"ID":"10bad8ad-782d-4a52-8450-c3399c47dc9b","Type":"ContainerStarted","Data":"c2d7769b67b643a510b18897d3bc09353693d5ade53952d75191f53e822e8a92"}
Sep 17 18:39:31.374487 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.374466    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fp6c4" event={"ID":"05ce87e8-6de7-4fd2-bf76-ecef629fc3b2","Type":"ContainerStarted","Data":"ab91f372b9ef41dc899fb9c8e986b51534871820a4bcc8a351ee5d786ee8e821"}
Sep 17 18:39:31.377217 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.377147    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd" event={"ID":"9d5360ef-43f6-4b49-88bb-67b77b161636","Type":"ContainerStarted","Data":"1071fab3ad3e1ab7e7ed02dd5f4329d5707efcbbb5c798108b9351563188c22a"}
Sep 17 18:39:31.377282 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.377220    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd" event={"ID":"9d5360ef-43f6-4b49-88bb-67b77b161636","Type":"ContainerStarted","Data":"a592e394a8e720cda68e47e8949d7af5f1fbe0ae3cb9511bd6ae61ea88a8946c"}
Sep 17 18:39:31.380642 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.377557    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd" event={"ID":"9d5360ef-43f6-4b49-88bb-67b77b161636","Type":"ContainerStarted","Data":"74aa7fcd8f59aa5f23a78f07e44ca78d8485d29b8f753281bf16ec96e26c3036"}
Sep 17 18:39:31.382228 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.382206    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb" event={"ID":"87018d8d-b113-4d82-875a-48918709e1f4","Type":"ContainerStarted","Data":"9afe33a412197a793e9625ed5aa932c8120605329a6209786862076f55f1ba40"}
Sep 17 18:39:31.386775 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.386486    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-194.ec2.internal" podStartSLOduration=20.386453913 podStartE2EDuration="20.386453913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:15.343538097 +0000 UTC m=+7.222506855" watchObservedRunningTime="2026-09-17 18:39:31.386453913 +0000 UTC m=+23.265422669"
Sep 17 18:39:31.387187 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.387146    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pdx8t" podStartSLOduration=3.700426191 podStartE2EDuration="21.387132542s" totalImagesPullingTime="17.686706351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:31.385131881 +0000 UTC m=+23.264100638" watchObservedRunningTime="2026-09-17 18:39:31.387132542 +0000 UTC m=+23.266101299"
Sep 17 18:39:31.464020 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.463979    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9nxm8" podStartSLOduration=11.250716847 podStartE2EDuration="21.463963789s" totalImagesPullingTime="10.213246942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:31.411952631 +0000 UTC m=+23.290921383" watchObservedRunningTime="2026-09-17 18:39:31.463963789 +0000 UTC m=+23.342932547"
Sep 17 18:39:31.464355 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.464334    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fp6c4" podStartSLOduration=3.745996931 podStartE2EDuration="21.464325791s" totalImagesPullingTime="17.71832886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:31.463809948 +0000 UTC m=+23.342778705" watchObservedRunningTime="2026-09-17 18:39:31.464325791 +0000 UTC m=+23.343294548"
Sep 17 18:39:31.518705 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.518618    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nngbm" podStartSLOduration=3.836001574 podStartE2EDuration="21.51860189s" totalImagesPullingTime="17.682600316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:31.48607635 +0000 UTC m=+23.365045106" watchObservedRunningTime="2026-09-17 18:39:31.51860189 +0000 UTC m=+23.397570647"
Sep 17 18:39:31.556872 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.556827    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-59srk" podStartSLOduration=3.879526777 podStartE2EDuration="21.556804811s" totalImagesPullingTime="17.677278034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:31.519056831 +0000 UTC m=+23.398025588" watchObservedRunningTime="2026-09-17 18:39:31.556804811 +0000 UTC m=+23.435773569"
Sep 17 18:39:31.595670 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.595638    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9nxm8"
Sep 17 18:39:31.596371 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:31.596350    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9nxm8"
Sep 17 18:39:32.084834 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:32.084800    2587 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 18:39:32.228599 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:32.228505    2587 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:39:32.084818642Z","UUID":"a10077d6-b084-4f93-adab-6f1c86bfb721","Handler":null,"Name":"","Endpoint":""}
Sep 17 18:39:32.230063 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:32.230031    2587 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 18:39:32.230063 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:32.230066    2587 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 18:39:32.252209 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:32.252182    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:32.252339 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:32.252319    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:32.252418 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:32.252393    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:32.252529 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:32.252503    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:32.252657 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:32.252633    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:32.252774 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:32.252697    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:32.388000 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:32.387964    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd" event={"ID":"9d5360ef-43f6-4b49-88bb-67b77b161636","Type":"ContainerStarted","Data":"5306834bdd1d41191932d5fae62762e0bd106f3d208d845f8fdfae18fd2482c9"}
Sep 17 18:39:32.388422 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:32.388053    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd" event={"ID":"9d5360ef-43f6-4b49-88bb-67b77b161636","Type":"ContainerStarted","Data":"bc3f206a5186672b9df94ac10d13d5e9965ed04f92a487e4e50248afc3e17bfa"}
Sep 17 18:39:32.388422 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:32.388104    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd" event={"ID":"9d5360ef-43f6-4b49-88bb-67b77b161636","Type":"ContainerStarted","Data":"725b5917e699cfcbe712f78153e7b51380292bded47de9367831711f5d576948"}
Sep 17 18:39:32.393239 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:32.393204    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb" event={"ID":"87018d8d-b113-4d82-875a-48918709e1f4","Type":"ContainerStarted","Data":"b741d5390551e6351ce83b23d0de9cdb1ae7c12b50e1bea09ac2519178fe3eff"}
Sep 17 18:39:32.393755 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:32.393735    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9nxm8"
Sep 17 18:39:32.394150 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:32.394130    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9nxm8"
Sep 17 18:39:33.396843 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:33.396804    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ms8k6" event={"ID":"aff06f71-dd37-479f-a73a-aaae682f8b78","Type":"ContainerStarted","Data":"68220bb7be1078460a75720da5af0f223ffb7cc19fa13c90f923a85bbf8c511d"}
Sep 17 18:39:33.399384 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:33.399351    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb" event={"ID":"87018d8d-b113-4d82-875a-48918709e1f4","Type":"ContainerStarted","Data":"2e27f93d5a4043001a3d3d00d6c04001ff92d95ecf514d14eab86ce7f7d60e44"}
Sep 17 18:39:33.431903 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:33.431859    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ms8k6" podStartSLOduration=5.754301831 podStartE2EDuration="23.431842233s" totalImagesPullingTime="17.677540402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:33.413183823 +0000 UTC m=+25.292152580" watchObservedRunningTime="2026-09-17 18:39:33.431842233 +0000 UTC m=+25.310810991"
Sep 17 18:39:33.432238 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:33.432202    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f7jsb" podStartSLOduration=4.216881834 podStartE2EDuration="23.432192065s" totalImagesPullingTime="19.215310231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:33.431980887 +0000 UTC m=+25.310949636" watchObservedRunningTime="2026-09-17 18:39:33.432192065 +0000 UTC m=+25.311160832"
Sep 17 18:39:34.252753 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:34.252697    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:34.252753 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:34.252741    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:34.253036 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:34.252821    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:34.253036 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:34.252890    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:34.253036 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:34.252964    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:34.253174 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:34.253078    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:34.406037 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:34.405990    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd" event={"ID":"9d5360ef-43f6-4b49-88bb-67b77b161636","Type":"ContainerStarted","Data":"bf6c9b9324141733d3d29cfa65089a21935e7884db7ac091c93635aebcf73cba"}
Sep 17 18:39:36.253382 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:36.253158    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:36.253986 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:36.253229    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:36.253986 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:36.253397    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:36.253986 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:36.253255    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:36.253986 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:36.253483    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:36.253986 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:36.253576    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:36.414384 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:36.414349    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd" event={"ID":"9d5360ef-43f6-4b49-88bb-67b77b161636","Type":"ContainerStarted","Data":"4f6e9032124bbaac24c2868b474ec635ea338ae7bf45e492a144b660da48b46a"}
Sep 17 18:39:36.414569 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:36.414527    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:36.414631 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:36.414591    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:36.414716 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:36.414631    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:36.416505 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:36.416482    2587 generic.go:356] "Generic (PLEG): container finished" podID="75b0ba53-5b75-493d-a60e-f479b53a0faa" containerID="b9cb674a8dbd8e9a9b403de0f502d1cf4e73529e93573f4bb30baf4d95ec1c3e" exitCode=0
Sep 17 18:39:36.416612 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:36.416519    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4t6b" event={"ID":"75b0ba53-5b75-493d-a60e-f479b53a0faa","Type":"ContainerDied","Data":"b9cb674a8dbd8e9a9b403de0f502d1cf4e73529e93573f4bb30baf4d95ec1c3e"}
Sep 17 18:39:36.435485 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:36.435460    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:36.435696 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:36.435679    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:39:36.453141 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:36.453093    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd" podStartSLOduration=8.747913427 podStartE2EDuration="26.453079349s" totalImagesPullingTime="17.705165922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:36.452747836 +0000 UTC m=+28.331716593" watchObservedRunningTime="2026-09-17 18:39:36.453079349 +0000 UTC m=+28.332048105"
Sep 17 18:39:38.144612 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:38.144425    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9sj77"]
Sep 17 18:39:38.145219 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:38.144746    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:38.145219 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:38.144865    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:38.149032 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:38.149006    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7frk4"]
Sep 17 18:39:38.149148 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:38.149133    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:38.149254 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:38.149235    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:38.153797 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:38.153769    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7m9j8"]
Sep 17 18:39:38.153920 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:38.153906    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:38.154076 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:38.154054    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:38.423398 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:38.423310    2587 generic.go:356] "Generic (PLEG): container finished" podID="75b0ba53-5b75-493d-a60e-f479b53a0faa" containerID="8dd9140bf21aa49c464255128862c3e03a3a2fbcedaf11dfe3402c32c0ba53a9" exitCode=0
Sep 17 18:39:38.423398 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:38.423380    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4t6b" event={"ID":"75b0ba53-5b75-493d-a60e-f479b53a0faa","Type":"ContainerDied","Data":"8dd9140bf21aa49c464255128862c3e03a3a2fbcedaf11dfe3402c32c0ba53a9"}
Sep 17 18:39:39.253055 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:39.253018    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:39.253535 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:39.253158    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:40.255533 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:40.255509    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:40.256017 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:40.255973    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:40.256277 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:40.256087    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:40.256277 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:40.256211    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:40.430350 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:40.430318    2587 generic.go:356] "Generic (PLEG): container finished" podID="75b0ba53-5b75-493d-a60e-f479b53a0faa" containerID="fb39053defa8eb194bd270bffa049e9a39b69dbf551b7f07795314a5784dece1" exitCode=0
Sep 17 18:39:40.430496 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:40.430373    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4t6b" event={"ID":"75b0ba53-5b75-493d-a60e-f479b53a0faa","Type":"ContainerDied","Data":"fb39053defa8eb194bd270bffa049e9a39b69dbf551b7f07795314a5784dece1"}
Sep 17 18:39:41.252386 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:41.252357    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:41.252531 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:41.252445    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:42.252795 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:42.252762    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:42.253262 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:42.252773    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:42.253262 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:42.252886    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sj77" podUID="ddd560fe-7729-4b3b-a307-d896e48b0baa"
Sep 17 18:39:42.253262 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:42.252971    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7m9j8" podUID="b425f704-e370-447b-a2f4-23d81057111d"
Sep 17 18:39:43.252423 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:43.252387    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:43.252711 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:43.252524    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7frk4" podUID="3821fdb9-ccfa-42c2-b476-9c624935e87e"
Sep 17 18:39:43.883115 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:43.883034    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs\") pod \"network-metrics-daemon-7m9j8\" (UID: \"b425f704-e370-447b-a2f4-23d81057111d\") " pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:43.883115 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:43.883082    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:43.883602 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:43.883175    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:43.883602 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:43.883176    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:43.883602 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:43.883225    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret podName:ddd560fe-7729-4b3b-a307-d896e48b0baa nodeName:}" failed. No retries permitted until 2026-09-17 18:40:15.883213013 +0000 UTC m=+67.762181762 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret") pod "global-pull-secret-syncer-9sj77" (UID: "ddd560fe-7729-4b3b-a307-d896e48b0baa") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:43.883602 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:43.883239    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs podName:b425f704-e370-447b-a2f4-23d81057111d nodeName:}" failed. No retries permitted until 2026-09-17 18:40:15.883232655 +0000 UTC m=+67.762201389 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs") pod "network-metrics-daemon-7m9j8" (UID: "b425f704-e370-447b-a2f4-23d81057111d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:44.085424 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.085386    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qws\" (UniqueName: \"kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws\") pod \"network-check-target-7frk4\" (UID: \"3821fdb9-ccfa-42c2-b476-9c624935e87e\") " pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:44.085591 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:44.085555    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:44.085591 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:44.085575    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:44.085591 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:44.085584    2587 projected.go:196] Error preparing data for projected volume kube-api-access-d5qws for pod openshift-network-diagnostics/network-check-target-7frk4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:44.085732 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:44.085648    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws podName:3821fdb9-ccfa-42c2-b476-9c624935e87e nodeName:}" failed. No retries permitted until 2026-09-17 18:40:16.085631193 +0000 UTC m=+67.964599931 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5qws" (UniqueName: "kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws") pod "network-check-target-7frk4" (UID: "3821fdb9-ccfa-42c2-b476-9c624935e87e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:44.095300 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.095278    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-194.ec2.internal" event="NodeReady"
Sep 17 18:39:44.095432 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.095405    2587 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 18:39:44.139700 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.139626    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5894f4d576-l4h7z"]
Sep 17 18:39:44.170592 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.170561    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9r5ml"]
Sep 17 18:39:44.170776 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.170750    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.175742 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.175719    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 18:39:44.175916 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.175751    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 18:39:44.177127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.177109    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-spfxb\""
Sep 17 18:39:44.177592 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.177566    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 18:39:44.184975 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.184957    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 18:39:44.186676 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.186645    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5894f4d576-l4h7z"]
Sep 17 18:39:44.186765 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.186682    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lv2sg"]
Sep 17 18:39:44.186809 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.186773    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:44.190534 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.190515    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 18:39:44.190918 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.190771    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gj7h2\""
Sep 17 18:39:44.190918 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.190846    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 18:39:44.204584 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.204550    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9r5ml"]
Sep 17 18:39:44.204703 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.204596    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lv2sg"]
Sep 17 18:39:44.204703 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.204637    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lv2sg"
Sep 17 18:39:44.208063 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.208042    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 18:39:44.208296 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.208280    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l4x2p\""
Sep 17 18:39:44.208584 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.208565    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 18:39:44.208674 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.208566    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 18:39:44.252934 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.252901    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:39:44.253126 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.252963    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:39:44.256169 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.256147    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:39:44.256334 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.256175    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:39:44.256645 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.256629    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7pg8w\""
Sep 17 18:39:44.287193 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.287156    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-ca-trust-extracted\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.287362 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.287201    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-certificates\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.287362 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.287251    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-installation-pull-secrets\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.287362 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.287280    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8j4p\" (UniqueName: \"kubernetes.io/projected/c65d126d-844f-4c22-b0c8-3747a48aa285-kube-api-access-c8j4p\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:44.287362 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.287342    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-bound-sa-token\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.287520 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.287396    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5378b81b-a0a2-4395-ab42-4faae5607cfc-cert\") pod \"ingress-canary-lv2sg\" (UID: \"5378b81b-a0a2-4395-ab42-4faae5607cfc\") " pod="openshift-ingress-canary/ingress-canary-lv2sg"
Sep 17 18:39:44.287520 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.287423    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.287520 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.287463    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c65d126d-844f-4c22-b0c8-3747a48aa285-config-volume\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:44.287520 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.287486    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b56mh\" (UniqueName: \"kubernetes.io/projected/5378b81b-a0a2-4395-ab42-4faae5607cfc-kube-api-access-b56mh\") pod \"ingress-canary-lv2sg\" (UID: \"5378b81b-a0a2-4395-ab42-4faae5607cfc\") " pod="openshift-ingress-canary/ingress-canary-lv2sg"
Sep 17 18:39:44.287520 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.287512    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqnfj\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-kube-api-access-cqnfj\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.287708 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.287563    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-trusted-ca\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.287708 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.287605    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c65d126d-844f-4c22-b0c8-3747a48aa285-tmp-dir\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:44.287708 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.287635    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:44.287708 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.287655    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-image-registry-private-configuration\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.292295 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.292244    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:44.388848 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.388802    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-ca-trust-extracted\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.388848 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.388854    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-certificates\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.389237 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.389218    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-installation-pull-secrets\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.389325 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.389247    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8j4p\" (UniqueName: \"kubernetes.io/projected/c65d126d-844f-4c22-b0c8-3747a48aa285-kube-api-access-c8j4p\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:44.389325 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.389267    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-bound-sa-token\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.389325 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.389304    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5378b81b-a0a2-4395-ab42-4faae5607cfc-cert\") pod \"ingress-canary-lv2sg\" (UID: \"5378b81b-a0a2-4395-ab42-4faae5607cfc\") " pod="openshift-ingress-canary/ingress-canary-lv2sg"
Sep 17 18:39:44.389325 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.389304    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-ca-trust-extracted\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.389509 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.389399    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-certificates\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.389509 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.389415    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.389509 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.389471    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c65d126d-844f-4c22-b0c8-3747a48aa285-config-volume\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:44.389509 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:44.389496    2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:44.389509 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:44.389512    2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5894f4d576-l4h7z: secret "image-registry-tls" not found
Sep 17 18:39:44.389758 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:44.389561    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls podName:5b9d9afe-fb1a-4eec-9e77-9e9675ea2421 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:44.88954527 +0000 UTC m=+36.768514021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls") pod "image-registry-5894f4d576-l4h7z" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421") : secret "image-registry-tls" not found
Sep 17 18:39:44.389758 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.389498    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b56mh\" (UniqueName: \"kubernetes.io/projected/5378b81b-a0a2-4395-ab42-4faae5607cfc-kube-api-access-b56mh\") pod \"ingress-canary-lv2sg\" (UID: \"5378b81b-a0a2-4395-ab42-4faae5607cfc\") " pod="openshift-ingress-canary/ingress-canary-lv2sg"
Sep 17 18:39:44.389758 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.389696    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqnfj\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-kube-api-access-cqnfj\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.389758 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.389727    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-trusted-ca\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.390020 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.389787    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c65d126d-844f-4c22-b0c8-3747a48aa285-tmp-dir\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:44.390020 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.389818    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:44.390020 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.389847    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-image-registry-private-configuration\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.390020 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:44.389912    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:44.390020 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:44.390010    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls podName:c65d126d-844f-4c22-b0c8-3747a48aa285 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:44.889989455 +0000 UTC m=+36.768958208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls") pod "dns-default-9r5ml" (UID: "c65d126d-844f-4c22-b0c8-3747a48aa285") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:44.390189 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.390094    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c65d126d-844f-4c22-b0c8-3747a48aa285-config-volume\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:44.391304 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.390277    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c65d126d-844f-4c22-b0c8-3747a48aa285-tmp-dir\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:44.392923 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.392760    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-trusted-ca\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.394064 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.394039    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-installation-pull-secrets\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.396160 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.395365    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-image-registry-private-configuration\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.396934 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.396910    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5378b81b-a0a2-4395-ab42-4faae5607cfc-cert\") pod \"ingress-canary-lv2sg\" (UID: \"5378b81b-a0a2-4395-ab42-4faae5607cfc\") " pod="openshift-ingress-canary/ingress-canary-lv2sg"
Sep 17 18:39:44.401257 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.401236    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8j4p\" (UniqueName: \"kubernetes.io/projected/c65d126d-844f-4c22-b0c8-3747a48aa285-kube-api-access-c8j4p\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:44.404853 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.404834    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqnfj\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-kube-api-access-cqnfj\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.404933 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.404882    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-bound-sa-token\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.405160 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.405142    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b56mh\" (UniqueName: \"kubernetes.io/projected/5378b81b-a0a2-4395-ab42-4faae5607cfc-kube-api-access-b56mh\") pod \"ingress-canary-lv2sg\" (UID: \"5378b81b-a0a2-4395-ab42-4faae5607cfc\") " pod="openshift-ingress-canary/ingress-canary-lv2sg"
Sep 17 18:39:44.422054 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.421859    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:44.514422 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.514389    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lv2sg"
Sep 17 18:39:44.523228 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.523186    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:44.823119 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.823094    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lv2sg"]
Sep 17 18:39:44.826295 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:39:44.826270    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5378b81b_a0a2_4395_ab42_4faae5607cfc.slice/crio-8ac7a8ea6c0f2999d1e6ea11b0466370d542b7e3bbb0c2490a3553d9c252a145 WatchSource:0}: Error finding container 8ac7a8ea6c0f2999d1e6ea11b0466370d542b7e3bbb0c2490a3553d9c252a145: Status 404 returned error can't find the container with id 8ac7a8ea6c0f2999d1e6ea11b0466370d542b7e3bbb0c2490a3553d9c252a145
Sep 17 18:39:44.893704 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.893676    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:44.894243 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:44.893746    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:44.894243 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:44.893847    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:44.894243 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:44.893903    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls podName:c65d126d-844f-4c22-b0c8-3747a48aa285 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:45.893889275 +0000 UTC m=+37.772858010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls") pod "dns-default-9r5ml" (UID: "c65d126d-844f-4c22-b0c8-3747a48aa285") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:44.894243 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:44.893850    2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:44.894243 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:44.893959    2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5894f4d576-l4h7z: secret "image-registry-tls" not found
Sep 17 18:39:44.894243 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:44.894011    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls podName:5b9d9afe-fb1a-4eec-9e77-9e9675ea2421 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:45.893987929 +0000 UTC m=+37.772956678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls") pod "image-registry-5894f4d576-l4h7z" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421") : secret "image-registry-tls" not found
Sep 17 18:39:45.231038 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:45.230934    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:45.252402 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:45.252370    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:39:45.255443 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:45.255420    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5jd5p\""
Sep 17 18:39:45.255711 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:45.255694    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:39:45.256533 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:45.256510    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:39:45.356854 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:45.356825    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:45.445450 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:45.445418    2587 generic.go:356] "Generic (PLEG): container finished" podID="75b0ba53-5b75-493d-a60e-f479b53a0faa" containerID="95d26683d8a9d2536e99e78d482dd930a6cc9df21b31c9bfed8278a9a96ccdc8" exitCode=0
Sep 17 18:39:45.445639 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:45.445503    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4t6b" event={"ID":"75b0ba53-5b75-493d-a60e-f479b53a0faa","Type":"ContainerDied","Data":"95d26683d8a9d2536e99e78d482dd930a6cc9df21b31c9bfed8278a9a96ccdc8"}
Sep 17 18:39:45.447122 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:45.447091    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lv2sg" event={"ID":"5378b81b-a0a2-4395-ab42-4faae5607cfc","Type":"ContainerStarted","Data":"8ac7a8ea6c0f2999d1e6ea11b0466370d542b7e3bbb0c2490a3553d9c252a145"}
Sep 17 18:39:45.474261 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:45.474234    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:45.567898 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:45.567424    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:45.903434 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:45.903352    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:45.903434 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:45.903423    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:45.903969 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:45.903516    2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:45.903969 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:45.903533    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:45.903969 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:45.903536    2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5894f4d576-l4h7z: secret "image-registry-tls" not found
Sep 17 18:39:45.903969 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:45.903597    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls podName:5b9d9afe-fb1a-4eec-9e77-9e9675ea2421 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:47.903582061 +0000 UTC m=+39.782550799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls") pod "image-registry-5894f4d576-l4h7z" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421") : secret "image-registry-tls" not found
Sep 17 18:39:45.903969 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:45.903616    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls podName:c65d126d-844f-4c22-b0c8-3747a48aa285 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:47.903606122 +0000 UTC m=+39.782574856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls") pod "dns-default-9r5ml" (UID: "c65d126d-844f-4c22-b0c8-3747a48aa285") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:46.224448 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:46.224362    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:46.294014 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:46.293985    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:46.453433 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:46.453398    2587 generic.go:356] "Generic (PLEG): container finished" podID="75b0ba53-5b75-493d-a60e-f479b53a0faa" containerID="bf3041995d079496840e0a2975345813f71a34a7f9f34e32db5deab0baf59a30" exitCode=0
Sep 17 18:39:46.453616 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:46.453442    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4t6b" event={"ID":"75b0ba53-5b75-493d-a60e-f479b53a0faa","Type":"ContainerDied","Data":"bf3041995d079496840e0a2975345813f71a34a7f9f34e32db5deab0baf59a30"}
Sep 17 18:39:46.613486 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:46.613454    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:46.714475 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:46.714435    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:47.460488 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:47.460291    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w4t6b" event={"ID":"75b0ba53-5b75-493d-a60e-f479b53a0faa","Type":"ContainerStarted","Data":"5715a250d6a095a23dd7d2018e44ba815b93700b711c965f7b343d8968049ee4"}
Sep 17 18:39:47.484176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:47.484110    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w4t6b" podStartSLOduration=8.955006143 podStartE2EDuration="37.484089012s" totalImagesPullingTime="28.529082869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:47.48274031 +0000 UTC m=+39.361709080" watchObservedRunningTime="2026-09-17 18:39:47.484089012 +0000 UTC m=+39.363057771"
Sep 17 18:39:47.608843 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:47.608794    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:47.711617 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:47.711586    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:47.811965 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:47.811899    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:47.915034 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:47.915004    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:47.920860 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:47.920830    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:47.920988 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:47.920894    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:47.921046 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:47.921007    2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:47.921046 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:47.921028    2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5894f4d576-l4h7z: secret "image-registry-tls" not found
Sep 17 18:39:47.921140 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:47.921075    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls podName:5b9d9afe-fb1a-4eec-9e77-9e9675ea2421 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:51.921060302 +0000 UTC m=+43.800029037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls") pod "image-registry-5894f4d576-l4h7z" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421") : secret "image-registry-tls" not found
Sep 17 18:39:47.921140 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:47.921007    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:47.921228 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:47.921157    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls podName:c65d126d-844f-4c22-b0c8-3747a48aa285 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:51.921137862 +0000 UTC m=+43.800106613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls") pod "dns-default-9r5ml" (UID: "c65d126d-844f-4c22-b0c8-3747a48aa285") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:48.464016 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:48.463974    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lv2sg" event={"ID":"5378b81b-a0a2-4395-ab42-4faae5607cfc","Type":"ContainerStarted","Data":"7e5644881a3f633aaaee45f5af326cc3ef34c31615ab3992c1e30fe2f75abab2"}
Sep 17 18:39:48.483008 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:48.482931    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lv2sg" podStartSLOduration=1.947383765 podStartE2EDuration="4.482914296s" totalImagesPullingTime="2.535530531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:48.481972418 +0000 UTC m=+40.360941174" watchObservedRunningTime="2026-09-17 18:39:48.482914296 +0000 UTC m=+40.361883053"
Sep 17 18:39:49.106184 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:49.106147    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:51.951776 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:51.951719    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:39:51.952302 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:51.951815    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:39:51.952302 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:51.951869    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:51.952302 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:51.951903    2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:51.952302 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:51.951913    2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5894f4d576-l4h7z: secret "image-registry-tls" not found
Sep 17 18:39:51.952302 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:51.951935    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls podName:c65d126d-844f-4c22-b0c8-3747a48aa285 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:59.9519196 +0000 UTC m=+51.830888335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls") pod "dns-default-9r5ml" (UID: "c65d126d-844f-4c22-b0c8-3747a48aa285") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:51.952302 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:39:51.951977    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls podName:5b9d9afe-fb1a-4eec-9e77-9e9675ea2421 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:59.951966201 +0000 UTC m=+51.830934936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls") pod "image-registry-5894f4d576-l4h7z" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421") : secret "image-registry-tls" not found
Sep 17 18:39:53.185573 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:53.185540    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:53.256503 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:39:53.256469    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:00.010754 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:00.010715    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:40:00.011260 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:00.010769    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:40:00.011260 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:00.010862    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:40:00.011260 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:00.010871    2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:40:00.011260 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:00.010892    2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5894f4d576-l4h7z: secret "image-registry-tls" not found
Sep 17 18:40:00.011260 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:00.010916    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls podName:c65d126d-844f-4c22-b0c8-3747a48aa285 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:16.010902323 +0000 UTC m=+67.889871061 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls") pod "dns-default-9r5ml" (UID: "c65d126d-844f-4c22-b0c8-3747a48aa285") : secret "dns-default-metrics-tls" not found
Sep 17 18:40:00.011260 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:00.010971    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls podName:5b9d9afe-fb1a-4eec-9e77-9e9675ea2421 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:16.010930321 +0000 UTC m=+67.889899091 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls") pod "image-registry-5894f4d576-l4h7z" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421") : secret "image-registry-tls" not found
Sep 17 18:40:02.783995 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:02.783963    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:02.854369 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:02.854310    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:03.724833 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:03.724786    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:03.824370 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:03.824335    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:04.288351 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.288320    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pc2jz"]
Sep 17 18:40:04.316977 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.316926    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pc2jz"]
Sep 17 18:40:04.317137 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.317046    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pc2jz"
Sep 17 18:40:04.320142 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.320117    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5bqd6\""
Sep 17 18:40:04.320930 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.320916    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 18:40:04.321010 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.320977    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:40:04.406204 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.406171    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k"]
Sep 17 18:40:04.426404 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.426377    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp"]
Sep 17 18:40:04.426584 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.426504    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k"
Sep 17 18:40:04.434158 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.434129    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 18:40:04.434314 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.434267    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 18:40:04.434492 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.434474    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wmldr\""
Sep 17 18:40:04.434704 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.434684    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:40:04.434802 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.434789    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 18:40:04.442373 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.442356    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"]
Sep 17 18:40:04.442477 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.442462    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp"
Sep 17 18:40:04.445232 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.445211    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 18:40:04.445370 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.445216    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:40:04.445880 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.445860    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 18:40:04.446642 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.446604    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 18:40:04.447577 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.447556    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-j75z7\""
Sep 17 18:40:04.447818 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.447795    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dszgn\" (UniqueName: \"kubernetes.io/projected/da763fad-c5e4-4748-8c41-bc1c8c896ef3-kube-api-access-dszgn\") pod \"volume-data-source-validator-5c4db8bdf9-pc2jz\" (UID: \"da763fad-c5e4-4748-8c41-bc1c8c896ef3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pc2jz"
Sep 17 18:40:04.456447 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.456425    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7646b475dd-4rb55"]
Sep 17 18:40:04.456566 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.456487    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:04.459132 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.459114    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 18:40:04.459432 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.459419    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 18:40:04.460793 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.460777    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7shh9\""
Sep 17 18:40:04.460886 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.460792    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 18:40:04.460886 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.460831    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 18:40:04.478463 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.478439    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-fm5ml"]
Sep 17 18:40:04.478623 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.478572    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:04.485009 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.484985    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tczk5\""
Sep 17 18:40:04.485149 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.485043    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 18:40:04.485149 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.485049    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 18:40:04.485279 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.485262    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 18:40:04.485397 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.485268    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 18:40:04.485397 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.485325    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 18:40:04.485544 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.485529    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 18:40:04.502611 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.502585    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"]
Sep 17 18:40:04.502611 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.502613    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp"]
Sep 17 18:40:04.502824 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.502627    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7646b475dd-4rb55"]
Sep 17 18:40:04.502824 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.502648    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k"]
Sep 17 18:40:04.502824 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.502667    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-fm5ml"]
Sep 17 18:40:04.502824 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.502700    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:04.504434 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.504414    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-dhhq4"]
Sep 17 18:40:04.508565 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.508538    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:40:04.508825 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.508803    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 18:40:04.508929 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.508814    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 18:40:04.508929 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.508853    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7kmkx\""
Sep 17 18:40:04.509260 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.509241    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 18:40:04.513162 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.513142    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 18:40:04.526828 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.526807    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dhhq4"
Sep 17 18:40:04.529622 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.529602    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-smnj5\""
Sep 17 18:40:04.530237 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.530219    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-dhhq4"]
Sep 17 18:40:04.549593 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.549528    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crkbr\" (UniqueName: \"kubernetes.io/projected/1ca5000b-7618-4602-8388-1c272aa30ae8-kube-api-access-crkbr\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:04.549700 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.549591    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7j4q\" (UniqueName: \"kubernetes.io/projected/1e4b5a1a-b85d-4ff5-9741-99678d16aad1-kube-api-access-k7j4q\") pod \"console-operator-fd4bb46c-fm5ml\" (UID: \"1e4b5a1a-b85d-4ff5-9741-99678d16aad1\") " pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:04.549700 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.549645    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e5a7245-d5c1-4419-9f7c-d3b5bdea88da-serving-cert\") pod \"service-ca-operator-57cdc4b768-f6j6k\" (UID: \"9e5a7245-d5c1-4419-9f7c-d3b5bdea88da\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k"
Sep 17 18:40:04.549700 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.549679    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dszgn\" (UniqueName: \"kubernetes.io/projected/da763fad-c5e4-4748-8c41-bc1c8c896ef3-kube-api-access-dszgn\") pod \"volume-data-source-validator-5c4db8bdf9-pc2jz\" (UID: \"da763fad-c5e4-4748-8c41-bc1c8c896ef3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pc2jz"
Sep 17 18:40:04.549867 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.549703    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e4b5a1a-b85d-4ff5-9741-99678d16aad1-trusted-ca\") pod \"console-operator-fd4bb46c-fm5ml\" (UID: \"1e4b5a1a-b85d-4ff5-9741-99678d16aad1\") " pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:04.549867 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.549805    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-stats-auth\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:04.549867 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.549850    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:04.550018 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.549875    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e5a7245-d5c1-4419-9f7c-d3b5bdea88da-config\") pod \"service-ca-operator-57cdc4b768-f6j6k\" (UID: \"9e5a7245-d5c1-4419-9f7c-d3b5bdea88da\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k"
Sep 17 18:40:04.550018 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.549901    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bklj\" (UniqueName: \"kubernetes.io/projected/ad67d410-cd73-46d3-81b8-d409d33ca41c-kube-api-access-6bklj\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:04.550018 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.549968    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e4b5a1a-b85d-4ff5-9741-99678d16aad1-serving-cert\") pod \"console-operator-fd4bb46c-fm5ml\" (UID: \"1e4b5a1a-b85d-4ff5-9741-99678d16aad1\") " pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:04.550018 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.549991    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ca5000b-7618-4602-8388-1c272aa30ae8-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:04.550157 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.550017    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwvkk\" (UniqueName: \"kubernetes.io/projected/9e5a7245-d5c1-4419-9f7c-d3b5bdea88da-kube-api-access-lwvkk\") pod \"service-ca-operator-57cdc4b768-f6j6k\" (UID: \"9e5a7245-d5c1-4419-9f7c-d3b5bdea88da\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k"
Sep 17 18:40:04.550157 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.550049    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lxkp\" (UniqueName: \"kubernetes.io/projected/3ba38230-b61f-4204-8b35-d60d7ee7b3b2-kube-api-access-5lxkp\") pod \"kube-storage-version-migrator-operator-55889f7b47-dn9jp\" (UID: \"3ba38230-b61f-4204-8b35-d60d7ee7b3b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp"
Sep 17 18:40:04.550157 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.550086    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e4b5a1a-b85d-4ff5-9741-99678d16aad1-config\") pod \"console-operator-fd4bb46c-fm5ml\" (UID: \"1e4b5a1a-b85d-4ff5-9741-99678d16aad1\") " pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:04.550157 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.550125    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kr6g\" (UniqueName: \"kubernetes.io/projected/eaae72fc-29f5-4005-bb56-032e7b7d4c78-kube-api-access-6kr6g\") pod \"network-check-source-7cbc45fddc-dhhq4\" (UID: \"eaae72fc-29f5-4005-bb56-032e7b7d4c78\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dhhq4"
Sep 17 18:40:04.550295 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.550158    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ba38230-b61f-4204-8b35-d60d7ee7b3b2-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-dn9jp\" (UID: \"3ba38230-b61f-4204-8b35-d60d7ee7b3b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp"
Sep 17 18:40:04.550295 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.550192    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-xdqjd"]
Sep 17 18:40:04.550369 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.550191    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ba38230-b61f-4204-8b35-d60d7ee7b3b2-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-dn9jp\" (UID: \"3ba38230-b61f-4204-8b35-d60d7ee7b3b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp"
Sep 17 18:40:04.550403 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.550372    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:04.550403 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.550396    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:04.550476 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.550422    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-default-certificate\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:04.570791 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.570759    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"]
Sep 17 18:40:04.570908 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.570836    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.575265 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.575246    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fj2rd\""
Sep 17 18:40:04.575364 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.575308    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 18:40:04.575767 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.575743    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 18:40:04.576344 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.576325    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 18:40:04.576654 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.576636    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 18:40:04.580792 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.580769    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dszgn\" (UniqueName: \"kubernetes.io/projected/da763fad-c5e4-4748-8c41-bc1c8c896ef3-kube-api-access-dszgn\") pod \"volume-data-source-validator-5c4db8bdf9-pc2jz\" (UID: \"da763fad-c5e4-4748-8c41-bc1c8c896ef3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pc2jz"
Sep 17 18:40:04.592809 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.592787    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"]
Sep 17 18:40:04.592916 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.592891    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"
Sep 17 18:40:04.593229 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.593197    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 18:40:04.596636 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.596621    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 18:40:04.596717 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.596665    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 18:40:04.596869 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.596854    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:40:04.611216 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.611195    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-xdqjd"]
Sep 17 18:40:04.626305 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.626281    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pc2jz"
Sep 17 18:40:04.628558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.628534    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9vvmx\""
Sep 17 18:40:04.650655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.650628    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crkbr\" (UniqueName: \"kubernetes.io/projected/1ca5000b-7618-4602-8388-1c272aa30ae8-kube-api-access-crkbr\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:04.650777 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.650668    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7j4q\" (UniqueName: \"kubernetes.io/projected/1e4b5a1a-b85d-4ff5-9741-99678d16aad1-kube-api-access-k7j4q\") pod \"console-operator-fd4bb46c-fm5ml\" (UID: \"1e4b5a1a-b85d-4ff5-9741-99678d16aad1\") " pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:04.650777 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.650702    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e5a7245-d5c1-4419-9f7c-d3b5bdea88da-serving-cert\") pod \"service-ca-operator-57cdc4b768-f6j6k\" (UID: \"9e5a7245-d5c1-4419-9f7c-d3b5bdea88da\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k"
Sep 17 18:40:04.650777 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.650723    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-serving-cert\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.650777 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.650746    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e4b5a1a-b85d-4ff5-9741-99678d16aad1-trusted-ca\") pod \"console-operator-fd4bb46c-fm5ml\" (UID: \"1e4b5a1a-b85d-4ff5-9741-99678d16aad1\") " pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:04.650777 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.650768    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-snapshots\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.651089 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.650833    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-stats-auth\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:04.651089 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651023    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:04.651089 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651059    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e5a7245-d5c1-4419-9f7c-d3b5bdea88da-config\") pod \"service-ca-operator-57cdc4b768-f6j6k\" (UID: \"9e5a7245-d5c1-4419-9f7c-d3b5bdea88da\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k"
Sep 17 18:40:04.651249 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651092    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bklj\" (UniqueName: \"kubernetes.io/projected/ad67d410-cd73-46d3-81b8-d409d33ca41c-kube-api-access-6bklj\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:04.651249 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651120    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e4b5a1a-b85d-4ff5-9741-99678d16aad1-serving-cert\") pod \"console-operator-fd4bb46c-fm5ml\" (UID: \"1e4b5a1a-b85d-4ff5-9741-99678d16aad1\") " pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:04.651249 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651146    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ca5000b-7618-4602-8388-1c272aa30ae8-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:04.651249 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651172    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwvkk\" (UniqueName: \"kubernetes.io/projected/9e5a7245-d5c1-4419-9f7c-d3b5bdea88da-kube-api-access-lwvkk\") pod \"service-ca-operator-57cdc4b768-f6j6k\" (UID: \"9e5a7245-d5c1-4419-9f7c-d3b5bdea88da\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k"
Sep 17 18:40:04.651249 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:04.651203    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle podName:ad67d410-cd73-46d3-81b8-d409d33ca41c nodeName:}" failed. No retries permitted until 2026-09-17 18:40:05.151181145 +0000 UTC m=+57.030149899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle") pod "router-default-7646b475dd-4rb55" (UID: "ad67d410-cd73-46d3-81b8-d409d33ca41c") : configmap references non-existent config key: service-ca.crt
Sep 17 18:40:04.651558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651247    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lxkp\" (UniqueName: \"kubernetes.io/projected/3ba38230-b61f-4204-8b35-d60d7ee7b3b2-kube-api-access-5lxkp\") pod \"kube-storage-version-migrator-operator-55889f7b47-dn9jp\" (UID: \"3ba38230-b61f-4204-8b35-d60d7ee7b3b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp"
Sep 17 18:40:04.651558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651279    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e4b5a1a-b85d-4ff5-9741-99678d16aad1-config\") pod \"console-operator-fd4bb46c-fm5ml\" (UID: \"1e4b5a1a-b85d-4ff5-9741-99678d16aad1\") " pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:04.651558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651307    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kr6g\" (UniqueName: \"kubernetes.io/projected/eaae72fc-29f5-4005-bb56-032e7b7d4c78-kube-api-access-6kr6g\") pod \"network-check-source-7cbc45fddc-dhhq4\" (UID: \"eaae72fc-29f5-4005-bb56-032e7b7d4c78\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dhhq4"
Sep 17 18:40:04.651558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651334    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-tmp\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.651558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651360    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.651558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651394    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqxbl\" (UniqueName: \"kubernetes.io/projected/b81db7d2-6a61-4665-8e55-f22e29e5e30f-kube-api-access-fqxbl\") pod \"cluster-samples-operator-685d7d68b5-77q28\" (UID: \"b81db7d2-6a61-4665-8e55-f22e29e5e30f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"
Sep 17 18:40:04.651558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651426    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ba38230-b61f-4204-8b35-d60d7ee7b3b2-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-dn9jp\" (UID: \"3ba38230-b61f-4204-8b35-d60d7ee7b3b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp"
Sep 17 18:40:04.651558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651458    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ba38230-b61f-4204-8b35-d60d7ee7b3b2-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-dn9jp\" (UID: \"3ba38230-b61f-4204-8b35-d60d7ee7b3b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp"
Sep 17 18:40:04.651558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651505    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqvt2\" (UniqueName: \"kubernetes.io/projected/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-kube-api-access-qqvt2\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.651558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651540    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77q28\" (UID: \"b81db7d2-6a61-4665-8e55-f22e29e5e30f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"
Sep 17 18:40:04.651558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651570    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:04.651558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651595    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:04.652308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.651624    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-default-certificate\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:04.653015 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.652306    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e5a7245-d5c1-4419-9f7c-d3b5bdea88da-config\") pod \"service-ca-operator-57cdc4b768-f6j6k\" (UID: \"9e5a7245-d5c1-4419-9f7c-d3b5bdea88da\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k"
Sep 17 18:40:04.653015 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.652429    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e4b5a1a-b85d-4ff5-9741-99678d16aad1-trusted-ca\") pod \"console-operator-fd4bb46c-fm5ml\" (UID: \"1e4b5a1a-b85d-4ff5-9741-99678d16aad1\") " pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:04.653015 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.652499    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.653015 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.652754    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ca5000b-7618-4602-8388-1c272aa30ae8-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:04.653015 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:04.652860    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:40:04.653015 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:04.652967    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs podName:ad67d410-cd73-46d3-81b8-d409d33ca41c nodeName:}" failed. No retries permitted until 2026-09-17 18:40:05.152917778 +0000 UTC m=+57.031886527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs") pod "router-default-7646b475dd-4rb55" (UID: "ad67d410-cd73-46d3-81b8-d409d33ca41c") : secret "router-metrics-certs-default" not found
Sep 17 18:40:04.653328 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:04.653218    2587 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:40:04.653328 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.653279    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e4b5a1a-b85d-4ff5-9741-99678d16aad1-config\") pod \"console-operator-fd4bb46c-fm5ml\" (UID: \"1e4b5a1a-b85d-4ff5-9741-99678d16aad1\") " pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:04.653328 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:04.653288    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls podName:1ca5000b-7618-4602-8388-1c272aa30ae8 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:05.153251181 +0000 UTC m=+57.032219935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qcr7l" (UID: "1ca5000b-7618-4602-8388-1c272aa30ae8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:40:04.653489 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.653467    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ba38230-b61f-4204-8b35-d60d7ee7b3b2-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-dn9jp\" (UID: \"3ba38230-b61f-4204-8b35-d60d7ee7b3b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp"
Sep 17 18:40:04.653705 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.653656    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e5a7245-d5c1-4419-9f7c-d3b5bdea88da-serving-cert\") pod \"service-ca-operator-57cdc4b768-f6j6k\" (UID: \"9e5a7245-d5c1-4419-9f7c-d3b5bdea88da\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k"
Sep 17 18:40:04.654374 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.654350    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e4b5a1a-b85d-4ff5-9741-99678d16aad1-serving-cert\") pod \"console-operator-fd4bb46c-fm5ml\" (UID: \"1e4b5a1a-b85d-4ff5-9741-99678d16aad1\") " pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:04.655526 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.655501    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ba38230-b61f-4204-8b35-d60d7ee7b3b2-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-dn9jp\" (UID: \"3ba38230-b61f-4204-8b35-d60d7ee7b3b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp"
Sep 17 18:40:04.657807 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.657785    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-default-certificate\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:04.664322 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.664294    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crkbr\" (UniqueName: \"kubernetes.io/projected/1ca5000b-7618-4602-8388-1c272aa30ae8-kube-api-access-crkbr\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:04.667006 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.666979    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-stats-auth\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:04.669895 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.669873    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bklj\" (UniqueName: \"kubernetes.io/projected/ad67d410-cd73-46d3-81b8-d409d33ca41c-kube-api-access-6bklj\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:04.670061 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.670036    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kr6g\" (UniqueName: \"kubernetes.io/projected/eaae72fc-29f5-4005-bb56-032e7b7d4c78-kube-api-access-6kr6g\") pod \"network-check-source-7cbc45fddc-dhhq4\" (UID: \"eaae72fc-29f5-4005-bb56-032e7b7d4c78\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dhhq4"
Sep 17 18:40:04.672128 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.672088    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7j4q\" (UniqueName: \"kubernetes.io/projected/1e4b5a1a-b85d-4ff5-9741-99678d16aad1-kube-api-access-k7j4q\") pod \"console-operator-fd4bb46c-fm5ml\" (UID: \"1e4b5a1a-b85d-4ff5-9741-99678d16aad1\") " pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:04.672875 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.672840    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lxkp\" (UniqueName: \"kubernetes.io/projected/3ba38230-b61f-4204-8b35-d60d7ee7b3b2-kube-api-access-5lxkp\") pod \"kube-storage-version-migrator-operator-55889f7b47-dn9jp\" (UID: \"3ba38230-b61f-4204-8b35-d60d7ee7b3b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp"
Sep 17 18:40:04.674526 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.674481    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwvkk\" (UniqueName: \"kubernetes.io/projected/9e5a7245-d5c1-4419-9f7c-d3b5bdea88da-kube-api-access-lwvkk\") pod \"service-ca-operator-57cdc4b768-f6j6k\" (UID: \"9e5a7245-d5c1-4419-9f7c-d3b5bdea88da\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k"
Sep 17 18:40:04.735329 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.735293    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k"
Sep 17 18:40:04.752573 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.752540    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp"
Sep 17 18:40:04.752850 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.752827    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-tmp\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.752850 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.752859    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.753077 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.752882    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqxbl\" (UniqueName: \"kubernetes.io/projected/b81db7d2-6a61-4665-8e55-f22e29e5e30f-kube-api-access-fqxbl\") pod \"cluster-samples-operator-685d7d68b5-77q28\" (UID: \"b81db7d2-6a61-4665-8e55-f22e29e5e30f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"
Sep 17 18:40:04.753077 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.752912    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqvt2\" (UniqueName: \"kubernetes.io/projected/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-kube-api-access-qqvt2\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.753077 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.752934    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77q28\" (UID: \"b81db7d2-6a61-4665-8e55-f22e29e5e30f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"
Sep 17 18:40:04.753077 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.752991    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.753971 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.753876    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.753971 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:04.753910    2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:40:04.754448 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.754169    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-serving-cert\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.754448 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.754243    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-snapshots\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.754448 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.754397    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-tmp\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.755431 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.754780    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.755431 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:04.754914    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls podName:b81db7d2-6a61-4665-8e55-f22e29e5e30f nodeName:}" failed. No retries permitted until 2026-09-17 18:40:05.254881482 +0000 UTC m=+57.133850223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-77q28" (UID: "b81db7d2-6a61-4665-8e55-f22e29e5e30f") : secret "samples-operator-tls" not found
Sep 17 18:40:04.758534 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.756396    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-snapshots\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.758534 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.756741    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:04.758534 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.757439    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-serving-cert\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.766689 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.766667    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqvt2\" (UniqueName: \"kubernetes.io/projected/2af1f706-2aa2-4b4a-9960-1eb443e3d40e-kube-api-access-qqvt2\") pod \"insights-operator-6f9c644dd9-xdqjd\" (UID: \"2af1f706-2aa2-4b4a-9960-1eb443e3d40e\") " pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.768424 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.768388    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqxbl\" (UniqueName: \"kubernetes.io/projected/b81db7d2-6a61-4665-8e55-f22e29e5e30f-kube-api-access-fqxbl\") pod \"cluster-samples-operator-685d7d68b5-77q28\" (UID: \"b81db7d2-6a61-4665-8e55-f22e29e5e30f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"
Sep 17 18:40:04.811506 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.811299    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:04.835423 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.835223    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dhhq4"
Sep 17 18:40:04.854073 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.853986    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:04.879901 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.879859    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd"
Sep 17 18:40:04.927172 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:04.927130    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pc2jz"]
Sep 17 18:40:04.930686 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:04.930654    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda763fad_c5e4_4748_8c41_bc1c8c896ef3.slice/crio-5ed59914092ace5f4540b2158d18447cd2011b503dbd1776ef7395b61279c3e1 WatchSource:0}: Error finding container 5ed59914092ace5f4540b2158d18447cd2011b503dbd1776ef7395b61279c3e1: Status 404 returned error can't find the container with id 5ed59914092ace5f4540b2158d18447cd2011b503dbd1776ef7395b61279c3e1
Sep 17 18:40:05.028176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.028144    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k"]
Sep 17 18:40:05.030731 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:05.030703    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5a7245_d5c1_4419_9f7c_d3b5bdea88da.slice/crio-43b618b164d3e5caa49a53e91acbc19b995d708b6a5623bb8f6819c04dfeac46 WatchSource:0}: Error finding container 43b618b164d3e5caa49a53e91acbc19b995d708b6a5623bb8f6819c04dfeac46: Status 404 returned error can't find the container with id 43b618b164d3e5caa49a53e91acbc19b995d708b6a5623bb8f6819c04dfeac46
Sep 17 18:40:05.061454 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.061380    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp"]
Sep 17 18:40:05.065087 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:05.065061    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba38230_b61f_4204_8b35_d60d7ee7b3b2.slice/crio-b4a3dcb286c90386c5581190dd60d5b370b74541b5bca5c1735b08821d785743 WatchSource:0}: Error finding container b4a3dcb286c90386c5581190dd60d5b370b74541b5bca5c1735b08821d785743: Status 404 returned error can't find the container with id b4a3dcb286c90386c5581190dd60d5b370b74541b5bca5c1735b08821d785743
Sep 17 18:40:05.105761 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.105734    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-fm5ml"]
Sep 17 18:40:05.109545 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:05.109514    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4b5a1a_b85d_4ff5_9741_99678d16aad1.slice/crio-c43db4cb05acf20778e2049239b0504e11d25daa0ec95b1ff6188a01c468a09a WatchSource:0}: Error finding container c43db4cb05acf20778e2049239b0504e11d25daa0ec95b1ff6188a01c468a09a: Status 404 returned error can't find the container with id c43db4cb05acf20778e2049239b0504e11d25daa0ec95b1ff6188a01c468a09a
Sep 17 18:40:05.138094 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.138073    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-dhhq4"]
Sep 17 18:40:05.140135 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:05.140109    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaae72fc_29f5_4005_bb56_032e7b7d4c78.slice/crio-0fa621198723b2297b0bac47ca166e0105b6d18e36ec81c7fa7d63cb6dcbf2b9 WatchSource:0}: Error finding container 0fa621198723b2297b0bac47ca166e0105b6d18e36ec81c7fa7d63cb6dcbf2b9: Status 404 returned error can't find the container with id 0fa621198723b2297b0bac47ca166e0105b6d18e36ec81c7fa7d63cb6dcbf2b9
Sep 17 18:40:05.158257 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.158200    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:05.158257 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.158232    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:05.158444 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.158325    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:05.158444 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:05.158375    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:40:05.158444 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:05.158440    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs podName:ad67d410-cd73-46d3-81b8-d409d33ca41c nodeName:}" failed. No retries permitted until 2026-09-17 18:40:06.158421398 +0000 UTC m=+58.037390150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs") pod "router-default-7646b475dd-4rb55" (UID: "ad67d410-cd73-46d3-81b8-d409d33ca41c") : secret "router-metrics-certs-default" not found
Sep 17 18:40:05.158571 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:05.158457    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle podName:ad67d410-cd73-46d3-81b8-d409d33ca41c nodeName:}" failed. No retries permitted until 2026-09-17 18:40:06.158448196 +0000 UTC m=+58.037416933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle") pod "router-default-7646b475dd-4rb55" (UID: "ad67d410-cd73-46d3-81b8-d409d33ca41c") : configmap references non-existent config key: service-ca.crt
Sep 17 18:40:05.158571 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:05.158470    2587 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:40:05.158571 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:05.158532    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls podName:1ca5000b-7618-4602-8388-1c272aa30ae8 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:06.158514256 +0000 UTC m=+58.037483015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qcr7l" (UID: "1ca5000b-7618-4602-8388-1c272aa30ae8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:40:05.176214 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.176182    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-xdqjd"]
Sep 17 18:40:05.178368 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:05.178343    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af1f706_2aa2_4b4a_9960_1eb443e3d40e.slice/crio-56b1382ae042601283515f0ff857df493f057fd284d3f3de2a082ab8b77bfbaa WatchSource:0}: Error finding container 56b1382ae042601283515f0ff857df493f057fd284d3f3de2a082ab8b77bfbaa: Status 404 returned error can't find the container with id 56b1382ae042601283515f0ff857df493f057fd284d3f3de2a082ab8b77bfbaa
Sep 17 18:40:05.258973 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.258902    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77q28\" (UID: \"b81db7d2-6a61-4665-8e55-f22e29e5e30f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"
Sep 17 18:40:05.259126 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:05.259088    2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:40:05.259173 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:05.259162    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls podName:b81db7d2-6a61-4665-8e55-f22e29e5e30f nodeName:}" failed. No retries permitted until 2026-09-17 18:40:06.259140054 +0000 UTC m=+58.138108790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-77q28" (UID: "b81db7d2-6a61-4665-8e55-f22e29e5e30f") : secret "samples-operator-tls" not found
Sep 17 18:40:05.507030 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.506970    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd" event={"ID":"2af1f706-2aa2-4b4a-9960-1eb443e3d40e","Type":"ContainerStarted","Data":"56b1382ae042601283515f0ff857df493f057fd284d3f3de2a082ab8b77bfbaa"}
Sep 17 18:40:05.508651 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.508612    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k" event={"ID":"9e5a7245-d5c1-4419-9f7c-d3b5bdea88da","Type":"ContainerStarted","Data":"43b618b164d3e5caa49a53e91acbc19b995d708b6a5623bb8f6819c04dfeac46"}
Sep 17 18:40:05.510005 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.509958    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dhhq4" event={"ID":"eaae72fc-29f5-4005-bb56-032e7b7d4c78","Type":"ContainerStarted","Data":"0fa621198723b2297b0bac47ca166e0105b6d18e36ec81c7fa7d63cb6dcbf2b9"}
Sep 17 18:40:05.512179 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.512135    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp" event={"ID":"3ba38230-b61f-4204-8b35-d60d7ee7b3b2","Type":"ContainerStarted","Data":"b4a3dcb286c90386c5581190dd60d5b370b74541b5bca5c1735b08821d785743"}
Sep 17 18:40:05.513448 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.513401    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml" event={"ID":"1e4b5a1a-b85d-4ff5-9741-99678d16aad1","Type":"ContainerStarted","Data":"c43db4cb05acf20778e2049239b0504e11d25daa0ec95b1ff6188a01c468a09a"}
Sep 17 18:40:05.514828 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:05.514805    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pc2jz" event={"ID":"da763fad-c5e4-4748-8c41-bc1c8c896ef3","Type":"ContainerStarted","Data":"5ed59914092ace5f4540b2158d18447cd2011b503dbd1776ef7395b61279c3e1"}
Sep 17 18:40:06.168338 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:06.168295    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:06.168788 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:06.168409    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:06.168788 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:06.168431    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:06.168788 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:06.168546    2587 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:40:06.168788 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:06.168603    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls podName:1ca5000b-7618-4602-8388-1c272aa30ae8 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:08.168584715 +0000 UTC m=+60.047553455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qcr7l" (UID: "1ca5000b-7618-4602-8388-1c272aa30ae8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:40:06.168788 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:06.168724    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle podName:ad67d410-cd73-46d3-81b8-d409d33ca41c nodeName:}" failed. No retries permitted until 2026-09-17 18:40:08.168711012 +0000 UTC m=+60.047679747 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle") pod "router-default-7646b475dd-4rb55" (UID: "ad67d410-cd73-46d3-81b8-d409d33ca41c") : configmap references non-existent config key: service-ca.crt
Sep 17 18:40:06.168788 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:06.168747    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:40:06.169154 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:06.168818    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs podName:ad67d410-cd73-46d3-81b8-d409d33ca41c nodeName:}" failed. No retries permitted until 2026-09-17 18:40:08.168800319 +0000 UTC m=+60.047769060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs") pod "router-default-7646b475dd-4rb55" (UID: "ad67d410-cd73-46d3-81b8-d409d33ca41c") : secret "router-metrics-certs-default" not found
Sep 17 18:40:06.270324 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:06.269656    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77q28\" (UID: \"b81db7d2-6a61-4665-8e55-f22e29e5e30f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"
Sep 17 18:40:06.270324 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:06.269873    2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:40:06.270324 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:06.269973    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls podName:b81db7d2-6a61-4665-8e55-f22e29e5e30f nodeName:}" failed. No retries permitted until 2026-09-17 18:40:08.269931463 +0000 UTC m=+60.148900220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-77q28" (UID: "b81db7d2-6a61-4665-8e55-f22e29e5e30f") : secret "samples-operator-tls" not found
Sep 17 18:40:06.520601 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:06.520558    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pc2jz" event={"ID":"da763fad-c5e4-4748-8c41-bc1c8c896ef3","Type":"ContainerStarted","Data":"9f107f4a22abdf3ff39d21e47427347fb78d5730b58c55f08be0b9009b1718f5"}
Sep 17 18:40:07.377597 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:07.377550    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:07.478668 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:07.478637    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:07.658614 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:07.658525    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:07.724922 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:07.724887    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:08.188249 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:08.188209    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:08.188449 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:08.188286    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:08.188449 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:08.188307    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:08.188449 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:08.188383    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle podName:ad67d410-cd73-46d3-81b8-d409d33ca41c nodeName:}" failed. No retries permitted until 2026-09-17 18:40:12.188361091 +0000 UTC m=+64.067329835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle") pod "router-default-7646b475dd-4rb55" (UID: "ad67d410-cd73-46d3-81b8-d409d33ca41c") : configmap references non-existent config key: service-ca.crt
Sep 17 18:40:08.188449 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:08.188406    2587 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:40:08.188449 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:08.188432    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:40:08.188449 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:08.188449    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls podName:1ca5000b-7618-4602-8388-1c272aa30ae8 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:12.188434797 +0000 UTC m=+64.067403549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qcr7l" (UID: "1ca5000b-7618-4602-8388-1c272aa30ae8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:40:08.188674 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:08.188483    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs podName:ad67d410-cd73-46d3-81b8-d409d33ca41c nodeName:}" failed. No retries permitted until 2026-09-17 18:40:12.188468745 +0000 UTC m=+64.067437507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs") pod "router-default-7646b475dd-4rb55" (UID: "ad67d410-cd73-46d3-81b8-d409d33ca41c") : secret "router-metrics-certs-default" not found
Sep 17 18:40:08.289734 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:08.289704    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77q28\" (UID: \"b81db7d2-6a61-4665-8e55-f22e29e5e30f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"
Sep 17 18:40:08.289905 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:08.289882    2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:40:08.290006 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:08.289969    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls podName:b81db7d2-6a61-4665-8e55-f22e29e5e30f nodeName:}" failed. No retries permitted until 2026-09-17 18:40:12.289934542 +0000 UTC m=+64.168903282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-77q28" (UID: "b81db7d2-6a61-4665-8e55-f22e29e5e30f") : secret "samples-operator-tls" not found
Sep 17 18:40:08.437673 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:08.437643    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzlnd"
Sep 17 18:40:08.494755 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:08.494703    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pc2jz" podStartSLOduration=3.046028224 podStartE2EDuration="4.494681737s" totalImagesPullingTime="1.448653513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:07.541020588 +0000 UTC m=+59.419989346" watchObservedRunningTime="2026-09-17 18:40:08.494681737 +0000 UTC m=+60.373650496"
Sep 17 18:40:08.527293 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:08.527255    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:08.643061 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:08.642925    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:08.735718 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:08.735679    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:08.839364 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:08.839276    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:08.946591 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:08.946558    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:09.294793 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:09.294753    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-872nz"]
Sep 17 18:40:09.297830 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:09.297805    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz"
Sep 17 18:40:09.301069 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:09.300975    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4bpj2\""
Sep 17 18:40:09.301069 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:09.300987    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 18:40:09.301438 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:09.301418    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 18:40:09.309251 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:09.309230    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-872nz"]
Sep 17 18:40:09.400314 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:09.400275    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/01a4765a-fe1b-40b2-a611-5e815c42cc84-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-872nz\" (UID: \"01a4765a-fe1b-40b2-a611-5e815c42cc84\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz"
Sep 17 18:40:09.400486 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:09.400450    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-872nz\" (UID: \"01a4765a-fe1b-40b2-a611-5e815c42cc84\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz"
Sep 17 18:40:09.501092 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:09.501059    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-872nz\" (UID: \"01a4765a-fe1b-40b2-a611-5e815c42cc84\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz"
Sep 17 18:40:09.501555 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:09.501123    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/01a4765a-fe1b-40b2-a611-5e815c42cc84-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-872nz\" (UID: \"01a4765a-fe1b-40b2-a611-5e815c42cc84\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz"
Sep 17 18:40:09.501555 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:09.501222    2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:40:09.501555 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:09.501311    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert podName:01a4765a-fe1b-40b2-a611-5e815c42cc84 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:10.001289249 +0000 UTC m=+61.880257989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-872nz" (UID: "01a4765a-fe1b-40b2-a611-5e815c42cc84") : secret "networking-console-plugin-cert" not found
Sep 17 18:40:09.501869 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:09.501842    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/01a4765a-fe1b-40b2-a611-5e815c42cc84-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-872nz\" (UID: \"01a4765a-fe1b-40b2-a611-5e815c42cc84\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz"
Sep 17 18:40:10.004702 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:10.004650    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-872nz\" (UID: \"01a4765a-fe1b-40b2-a611-5e815c42cc84\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz"
Sep 17 18:40:10.004887 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:10.004836    2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:40:10.004930 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:10.004916    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert podName:01a4765a-fe1b-40b2-a611-5e815c42cc84 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:11.004898429 +0000 UTC m=+62.883867168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-872nz" (UID: "01a4765a-fe1b-40b2-a611-5e815c42cc84") : secret "networking-console-plugin-cert" not found
Sep 17 18:40:10.537030 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:10.536586    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dhhq4" event={"ID":"eaae72fc-29f5-4005-bb56-032e7b7d4c78","Type":"ContainerStarted","Data":"e809856d0b8247204b0e3e84e284f2a024fb928595c9c67fd177bad61b26bdcc"}
Sep 17 18:40:10.538439 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:10.538408    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp" event={"ID":"3ba38230-b61f-4204-8b35-d60d7ee7b3b2","Type":"ContainerStarted","Data":"ec6dc94b7ec251f3e9f2092f040086328c1e808267609020de12850f4b9a0c07"}
Sep 17 18:40:10.540762 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:10.540740    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/0.log"
Sep 17 18:40:10.540879 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:10.540782    2587 generic.go:356] "Generic (PLEG): container finished" podID="1e4b5a1a-b85d-4ff5-9741-99678d16aad1" containerID="6dc6b17f9947dbf4fe07279d55bd31b8ec8efb5b9d8c143210861ca6e632b16a" exitCode=255
Sep 17 18:40:10.540879 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:10.540871    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml" event={"ID":"1e4b5a1a-b85d-4ff5-9741-99678d16aad1","Type":"ContainerDied","Data":"6dc6b17f9947dbf4fe07279d55bd31b8ec8efb5b9d8c143210861ca6e632b16a"}
Sep 17 18:40:10.541099 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:10.541081    2587 scope.go:117] "RemoveContainer" containerID="6dc6b17f9947dbf4fe07279d55bd31b8ec8efb5b9d8c143210861ca6e632b16a"
Sep 17 18:40:10.542988 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:10.542800    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd" event={"ID":"2af1f706-2aa2-4b4a-9960-1eb443e3d40e","Type":"ContainerStarted","Data":"9b49f88b3012aaaa60e90ca44905ff9dea7cdcf7a65a6038b8d13574e8f86a49"}
Sep 17 18:40:10.545694 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:10.545669    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k" event={"ID":"9e5a7245-d5c1-4419-9f7c-d3b5bdea88da","Type":"ContainerStarted","Data":"2f5db043d4d4db24985e1c7845ce1006302508bb8543e211d3cb8e4e5c6fb62d"}
Sep 17 18:40:10.562512 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:10.562473    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:10.562822 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:10.562777    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dhhq4" podStartSLOduration=1.446127621 podStartE2EDuration="6.562761608s" totalImagesPullingTime="5.116633987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:10.56228512 +0000 UTC m=+62.441253877" watchObservedRunningTime="2026-09-17 18:40:10.562761608 +0000 UTC m=+62.441730359"
Sep 17 18:40:10.582761 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:10.582704    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp" podStartSLOduration=1.405769244 podStartE2EDuration="6.582684112s" totalImagesPullingTime="5.176914868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:10.58189084 +0000 UTC m=+62.460859600" watchObservedRunningTime="2026-09-17 18:40:10.582684112 +0000 UTC m=+62.461652870"
Sep 17 18:40:10.633211 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:10.633076    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k" podStartSLOduration=1.407078958 podStartE2EDuration="6.63305404s" totalImagesPullingTime="5.225975082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:10.632397865 +0000 UTC m=+62.511366622" watchObservedRunningTime="2026-09-17 18:40:10.63305404 +0000 UTC m=+62.512022799"
Sep 17 18:40:11.005830 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.005795    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:11.014889 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.014857    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-872nz\" (UID: \"01a4765a-fe1b-40b2-a611-5e815c42cc84\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz"
Sep 17 18:40:11.015077 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:11.015056    2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:40:11.015149 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:11.015138    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert podName:01a4765a-fe1b-40b2-a611-5e815c42cc84 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:13.015116895 +0000 UTC m=+64.894085635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-872nz" (UID: "01a4765a-fe1b-40b2-a611-5e815c42cc84") : secret "networking-console-plugin-cert" not found
Sep 17 18:40:11.108154 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.108118    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:11.208766 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.208728    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:11.285589 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.285550    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:11.433354 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.433312    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:11.550887 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.550801    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 18:40:11.551599 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.551579    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/0.log"
Sep 17 18:40:11.551744 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.551623    2587 generic.go:356] "Generic (PLEG): container finished" podID="1e4b5a1a-b85d-4ff5-9741-99678d16aad1" containerID="b8c84218bc1ed393e39031a8118406fdb89d0a4840457a871800f954e4829ed6" exitCode=255
Sep 17 18:40:11.552542 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.552518    2587 scope.go:117] "RemoveContainer" containerID="b8c84218bc1ed393e39031a8118406fdb89d0a4840457a871800f954e4829ed6"
Sep 17 18:40:11.552721 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:11.552678    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-fm5ml_openshift-console-operator(1e4b5a1a-b85d-4ff5-9741-99678d16aad1)\"" pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml" podUID="1e4b5a1a-b85d-4ff5-9741-99678d16aad1"
Sep 17 18:40:11.553032 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.552874    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml" event={"ID":"1e4b5a1a-b85d-4ff5-9741-99678d16aad1","Type":"ContainerDied","Data":"b8c84218bc1ed393e39031a8118406fdb89d0a4840457a871800f954e4829ed6"}
Sep 17 18:40:11.553032 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.552928    2587 scope.go:117] "RemoveContainer" containerID="6dc6b17f9947dbf4fe07279d55bd31b8ec8efb5b9d8c143210861ca6e632b16a"
Sep 17 18:40:11.589121 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.589060    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd" podStartSLOduration=2.522660159 podStartE2EDuration="7.589037516s" totalImagesPullingTime="5.066377357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:10.655906097 +0000 UTC m=+62.534874859" watchObservedRunningTime="2026-09-17 18:40:11.589037516 +0000 UTC m=+63.468006274"
Sep 17 18:40:11.717480 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.717441    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:11.859265 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.859154    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:11.918673 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.918641    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ph9zh"]
Sep 17 18:40:11.921636 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.921614    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:11.935095 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.935070    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 18:40:11.935247 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.935123    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 18:40:11.935760 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.935601    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kzdh4\""
Sep 17 18:40:11.938959 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.938917    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ph9zh"]
Sep 17 18:40:11.981502 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:11.981451    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:12.025412 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.025372    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/857bafdf-f63a-4616-bb74-dfefd8f63fdf-data-volume\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.025612 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.025532    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/857bafdf-f63a-4616-bb74-dfefd8f63fdf-crio-socket\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.025684 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.025623    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.025684 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.025665    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zblgg\" (UniqueName: \"kubernetes.io/projected/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-api-access-zblgg\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.025773 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.025728    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.064364 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.064305    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:12.126694 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.126604    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/857bafdf-f63a-4616-bb74-dfefd8f63fdf-data-volume\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.126829 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.126728    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/857bafdf-f63a-4616-bb74-dfefd8f63fdf-crio-socket\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.126829 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.126779    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.126829 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.126797    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/857bafdf-f63a-4616-bb74-dfefd8f63fdf-crio-socket\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.126995 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.126806    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zblgg\" (UniqueName: \"kubernetes.io/projected/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-api-access-zblgg\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.126995 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.126899    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.126995 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.126926    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:40:12.127132 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.127023    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm podName:857bafdf-f63a-4616-bb74-dfefd8f63fdf nodeName:}" failed. No retries permitted until 2026-09-17 18:40:12.627005903 +0000 UTC m=+64.505974640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ph9zh" (UID: "857bafdf-f63a-4616-bb74-dfefd8f63fdf") : configmap "kube-rbac-proxy" not found
Sep 17 18:40:12.127132 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.127028    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/857bafdf-f63a-4616-bb74-dfefd8f63fdf-data-volume\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.127132 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.127092    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:12.127254 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.127155    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls podName:857bafdf-f63a-4616-bb74-dfefd8f63fdf nodeName:}" failed. No retries permitted until 2026-09-17 18:40:12.627137413 +0000 UTC m=+64.506106170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ph9zh" (UID: "857bafdf-f63a-4616-bb74-dfefd8f63fdf") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:12.137383 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.137348    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zblgg\" (UniqueName: \"kubernetes.io/projected/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-api-access-zblgg\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.227925 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.227880    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:12.228149 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.228013    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:12.228149 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.228039    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:12.228149 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.228076    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle podName:ad67d410-cd73-46d3-81b8-d409d33ca41c nodeName:}" failed. No retries permitted until 2026-09-17 18:40:20.228058735 +0000 UTC m=+72.107027499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle") pod "router-default-7646b475dd-4rb55" (UID: "ad67d410-cd73-46d3-81b8-d409d33ca41c") : configmap references non-existent config key: service-ca.crt
Sep 17 18:40:12.228149 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.228136    2587 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:40:12.228360 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.228152    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:40:12.228360 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.228191    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls podName:1ca5000b-7618-4602-8388-1c272aa30ae8 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:20.228176702 +0000 UTC m=+72.107145459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qcr7l" (UID: "1ca5000b-7618-4602-8388-1c272aa30ae8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:40:12.228360 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.228250    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs podName:ad67d410-cd73-46d3-81b8-d409d33ca41c nodeName:}" failed. No retries permitted until 2026-09-17 18:40:20.228238841 +0000 UTC m=+72.107207579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs") pod "router-default-7646b475dd-4rb55" (UID: "ad67d410-cd73-46d3-81b8-d409d33ca41c") : secret "router-metrics-certs-default" not found
Sep 17 18:40:12.329352 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.329320    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77q28\" (UID: \"b81db7d2-6a61-4665-8e55-f22e29e5e30f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"
Sep 17 18:40:12.329537 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.329475    2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:40:12.329579 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.329545    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls podName:b81db7d2-6a61-4665-8e55-f22e29e5e30f nodeName:}" failed. No retries permitted until 2026-09-17 18:40:20.329527833 +0000 UTC m=+72.208496590 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-77q28" (UID: "b81db7d2-6a61-4665-8e55-f22e29e5e30f") : secret "samples-operator-tls" not found
Sep 17 18:40:12.556450 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.556421    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 18:40:12.556868 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.556772    2587 scope.go:117] "RemoveContainer" containerID="b8c84218bc1ed393e39031a8118406fdb89d0a4840457a871800f954e4829ed6"
Sep 17 18:40:12.556959 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.556921    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-fm5ml_openshift-console-operator(1e4b5a1a-b85d-4ff5-9741-99678d16aad1)\"" pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml" podUID="1e4b5a1a-b85d-4ff5-9741-99678d16aad1"
Sep 17 18:40:12.631847 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.631802    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.632044 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.631883    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:12.632044 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.631960    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:40:12.632044 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.632004    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:12.632183 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.632052    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm podName:857bafdf-f63a-4616-bb74-dfefd8f63fdf nodeName:}" failed. No retries permitted until 2026-09-17 18:40:13.632031158 +0000 UTC m=+65.510999899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ph9zh" (UID: "857bafdf-f63a-4616-bb74-dfefd8f63fdf") : configmap "kube-rbac-proxy" not found
Sep 17 18:40:12.632183 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:12.632071    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls podName:857bafdf-f63a-4616-bb74-dfefd8f63fdf nodeName:}" failed. No retries permitted until 2026-09-17 18:40:13.632062151 +0000 UTC m=+65.511030893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ph9zh" (UID: "857bafdf-f63a-4616-bb74-dfefd8f63fdf") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:12.897506 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.897433    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7p56c"]
Sep 17 18:40:12.899541 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.899515    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7p56c"
Sep 17 18:40:12.910493 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.910470    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4bwkb\""
Sep 17 18:40:12.912099 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.912080    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 18:40:12.912210 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.912079    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 18:40:12.922614 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.922588    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7p56c"]
Sep 17 18:40:12.934839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.934805    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlkbk\" (UniqueName: \"kubernetes.io/projected/987d9911-68e0-478a-a60d-234fbb2fbdc1-kube-api-access-xlkbk\") pod \"migrator-76794cbd6c-7p56c\" (UID: \"987d9911-68e0-478a-a60d-234fbb2fbdc1\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7p56c"
Sep 17 18:40:12.974339 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:12.974304    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:13.035966 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.035906    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-872nz\" (UID: \"01a4765a-fe1b-40b2-a611-5e815c42cc84\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz"
Sep 17 18:40:13.036146 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.035988    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlkbk\" (UniqueName: \"kubernetes.io/projected/987d9911-68e0-478a-a60d-234fbb2fbdc1-kube-api-access-xlkbk\") pod \"migrator-76794cbd6c-7p56c\" (UID: \"987d9911-68e0-478a-a60d-234fbb2fbdc1\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7p56c"
Sep 17 18:40:13.036146 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:13.036103    2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:40:13.036235 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:13.036170    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert podName:01a4765a-fe1b-40b2-a611-5e815c42cc84 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:17.036154228 +0000 UTC m=+68.915122971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-872nz" (UID: "01a4765a-fe1b-40b2-a611-5e815c42cc84") : secret "networking-console-plugin-cert" not found
Sep 17 18:40:13.045252 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.045220    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nngbm_10bad8ad-782d-4a52-8450-c3399c47dc9b/dns-node-resolver/0.log"
Sep 17 18:40:13.047015 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.046995    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlkbk\" (UniqueName: \"kubernetes.io/projected/987d9911-68e0-478a-a60d-234fbb2fbdc1-kube-api-access-xlkbk\") pod \"migrator-76794cbd6c-7p56c\" (UID: \"987d9911-68e0-478a-a60d-234fbb2fbdc1\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7p56c"
Sep 17 18:40:13.074425 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.074391    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:13.208903 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.208819    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7p56c"
Sep 17 18:40:13.482237 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.482156    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-7p56c"]
Sep 17 18:40:13.484783 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:13.484757    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987d9911_68e0_478a_a60d_234fbb2fbdc1.slice/crio-4d99c25443ccff1cd519d82a0ef34d18ed516addc91f16e734f574e2f38e31ae WatchSource:0}: Error finding container 4d99c25443ccff1cd519d82a0ef34d18ed516addc91f16e734f574e2f38e31ae: Status 404 returned error can't find the container with id 4d99c25443ccff1cd519d82a0ef34d18ed516addc91f16e734f574e2f38e31ae
Sep 17 18:40:13.531517 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.531481    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-t7lq2"]
Sep 17 18:40:13.534077 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.534060    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-t7lq2"
Sep 17 18:40:13.537832 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.537805    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 18:40:13.538094 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.538073    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 18:40:13.538734 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.538721    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xcz8x\""
Sep 17 18:40:13.538789 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.538723    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 18:40:13.539003 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.538986    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 18:40:13.560351 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.560314    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7p56c" event={"ID":"987d9911-68e0-478a-a60d-234fbb2fbdc1","Type":"ContainerStarted","Data":"4d99c25443ccff1cd519d82a0ef34d18ed516addc91f16e734f574e2f38e31ae"}
Sep 17 18:40:13.562481 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.562454    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-t7lq2"]
Sep 17 18:40:13.610075 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.610041    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:13.641152 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.641118    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ef8cd742-7237-4c76-a079-d4f7c30a6632-signing-cabundle\") pod \"service-ca-55f6fc7b86-t7lq2\" (UID: \"ef8cd742-7237-4c76-a079-d4f7c30a6632\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t7lq2"
Sep 17 18:40:13.641358 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.641335    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ef8cd742-7237-4c76-a079-d4f7c30a6632-signing-key\") pod \"service-ca-55f6fc7b86-t7lq2\" (UID: \"ef8cd742-7237-4c76-a079-d4f7c30a6632\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t7lq2"
Sep 17 18:40:13.641426 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.641379    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp4bb\" (UniqueName: \"kubernetes.io/projected/ef8cd742-7237-4c76-a079-d4f7c30a6632-kube-api-access-bp4bb\") pod \"service-ca-55f6fc7b86-t7lq2\" (UID: \"ef8cd742-7237-4c76-a079-d4f7c30a6632\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t7lq2"
Sep 17 18:40:13.641479 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.641433    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:13.641529 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.641506    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:13.641577 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:13.641525    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:40:13.641626 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:13.641586    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm podName:857bafdf-f63a-4616-bb74-dfefd8f63fdf nodeName:}" failed. No retries permitted until 2026-09-17 18:40:15.641566634 +0000 UTC m=+67.520535388 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ph9zh" (UID: "857bafdf-f63a-4616-bb74-dfefd8f63fdf") : configmap "kube-rbac-proxy" not found
Sep 17 18:40:13.641683 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:13.641627    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:13.641683 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:13.641682    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls podName:857bafdf-f63a-4616-bb74-dfefd8f63fdf nodeName:}" failed. No retries permitted until 2026-09-17 18:40:15.641670635 +0000 UTC m=+67.520639370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ph9zh" (UID: "857bafdf-f63a-4616-bb74-dfefd8f63fdf") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:13.709906 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.709876    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:13.742104 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.742067    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ef8cd742-7237-4c76-a079-d4f7c30a6632-signing-cabundle\") pod \"service-ca-55f6fc7b86-t7lq2\" (UID: \"ef8cd742-7237-4c76-a079-d4f7c30a6632\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t7lq2"
Sep 17 18:40:13.742301 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.742240    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ef8cd742-7237-4c76-a079-d4f7c30a6632-signing-key\") pod \"service-ca-55f6fc7b86-t7lq2\" (UID: \"ef8cd742-7237-4c76-a079-d4f7c30a6632\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t7lq2"
Sep 17 18:40:13.742301 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.742272    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp4bb\" (UniqueName: \"kubernetes.io/projected/ef8cd742-7237-4c76-a079-d4f7c30a6632-kube-api-access-bp4bb\") pod \"service-ca-55f6fc7b86-t7lq2\" (UID: \"ef8cd742-7237-4c76-a079-d4f7c30a6632\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t7lq2"
Sep 17 18:40:13.742702 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.742684    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ef8cd742-7237-4c76-a079-d4f7c30a6632-signing-cabundle\") pod \"service-ca-55f6fc7b86-t7lq2\" (UID: \"ef8cd742-7237-4c76-a079-d4f7c30a6632\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t7lq2"
Sep 17 18:40:13.744800 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.744768    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ef8cd742-7237-4c76-a079-d4f7c30a6632-signing-key\") pod \"service-ca-55f6fc7b86-t7lq2\" (UID: \"ef8cd742-7237-4c76-a079-d4f7c30a6632\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t7lq2"
Sep 17 18:40:13.751265 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.751237    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp4bb\" (UniqueName: \"kubernetes.io/projected/ef8cd742-7237-4c76-a079-d4f7c30a6632-kube-api-access-bp4bb\") pod \"service-ca-55f6fc7b86-t7lq2\" (UID: \"ef8cd742-7237-4c76-a079-d4f7c30a6632\") " pod="openshift-service-ca/service-ca-55f6fc7b86-t7lq2"
Sep 17 18:40:13.841825 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.841799    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-59srk_0010ab57-08d7-481d-aa7f-7c480d8ece9f/node-ca/0.log"
Sep 17 18:40:13.842869 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:13.842847    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-t7lq2"
Sep 17 18:40:14.114844 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:14.114819    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8cd742_7237_4c76_a079_d4f7c30a6632.slice/crio-d55a66c423a3a8566dc7e4d181e7086e8b5799b4a471611a0996c64daf47d1ce WatchSource:0}: Error finding container d55a66c423a3a8566dc7e4d181e7086e8b5799b4a471611a0996c64daf47d1ce: Status 404 returned error can't find the container with id d55a66c423a3a8566dc7e4d181e7086e8b5799b4a471611a0996c64daf47d1ce
Sep 17 18:40:14.121584 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:14.120310    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-t7lq2"]
Sep 17 18:40:14.565643 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:14.565461    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-t7lq2" event={"ID":"ef8cd742-7237-4c76-a079-d4f7c30a6632","Type":"ContainerStarted","Data":"932fa155eda749df3c753e21cb2c245d923fbfe8a58a288b383fe29ddbb3a54d"}
Sep 17 18:40:14.565643 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:14.565526    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-t7lq2" event={"ID":"ef8cd742-7237-4c76-a079-d4f7c30a6632","Type":"ContainerStarted","Data":"d55a66c423a3a8566dc7e4d181e7086e8b5799b4a471611a0996c64daf47d1ce"}
Sep 17 18:40:14.585823 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:14.585763    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-t7lq2" podStartSLOduration=1.585741324 podStartE2EDuration="1.585741324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:14.584431427 +0000 UTC m=+66.463400186" watchObservedRunningTime="2026-09-17 18:40:14.585741324 +0000 UTC m=+66.464710104"
Sep 17 18:40:14.646549 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:14.646522    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lv2sg_5378b81b-a0a2-4395-ab42-4faae5607cfc/serve-healthcheck-canary/0.log"
Sep 17 18:40:14.699576 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:14.699516    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:14.811616 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:14.811580    2587 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:14.811788 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:14.811639    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:14.811990 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:14.811977    2587 scope.go:117] "RemoveContainer" containerID="b8c84218bc1ed393e39031a8118406fdb89d0a4840457a871800f954e4829ed6"
Sep 17 18:40:14.812145 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:14.812128    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-fm5ml_openshift-console-operator(1e4b5a1a-b85d-4ff5-9741-99678d16aad1)\"" pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml" podUID="1e4b5a1a-b85d-4ff5-9741-99678d16aad1"
Sep 17 18:40:14.954763 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:14.954651    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:15.111408 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.111364    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:15.521971 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.521860    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:15.571772 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.571733    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7p56c" event={"ID":"987d9911-68e0-478a-a60d-234fbb2fbdc1","Type":"ContainerStarted","Data":"62ec062cc7d317b8ab85c2ea5d3588bd7d35de5b5ded9037f05976dc1a673968"}
Sep 17 18:40:15.572276 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.571811    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7p56c" event={"ID":"987d9911-68e0-478a-a60d-234fbb2fbdc1","Type":"ContainerStarted","Data":"9f247f20c7e9d966f5c129f96a137e05b8ad04d225e4af9189aae73331917b6a"}
Sep 17 18:40:15.600472 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.600405    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-7p56c" podStartSLOduration=2.417479093 podStartE2EDuration="3.600383094s" totalImagesPullingTime="1.182904001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:15.599974427 +0000 UTC m=+67.478943188" watchObservedRunningTime="2026-09-17 18:40:15.600383094 +0000 UTC m=+67.479351853"
Sep 17 18:40:15.643646 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.643609    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:15.660961 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.660912    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:15.661110 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.660984    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:15.661293 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:15.661272    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:40:15.661450 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:15.661329    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:15.661506 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:15.661433    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm podName:857bafdf-f63a-4616-bb74-dfefd8f63fdf nodeName:}" failed. No retries permitted until 2026-09-17 18:40:19.661413241 +0000 UTC m=+71.540381978 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ph9zh" (UID: "857bafdf-f63a-4616-bb74-dfefd8f63fdf") : configmap "kube-rbac-proxy" not found
Sep 17 18:40:15.661568 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:15.661516    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls podName:857bafdf-f63a-4616-bb74-dfefd8f63fdf nodeName:}" failed. No retries permitted until 2026-09-17 18:40:19.661498071 +0000 UTC m=+71.540466820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ph9zh" (UID: "857bafdf-f63a-4616-bb74-dfefd8f63fdf") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:15.808497 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.808413    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:15.964048 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.963997    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:40:15.964248 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.964213    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs\") pod \"network-metrics-daemon-7m9j8\" (UID: \"b425f704-e370-447b-a2f4-23d81057111d\") " pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:40:15.968660 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.968625    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:40:15.968854 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.968823    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:40:15.974801 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:15.974779    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 18:40:15.974901 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:15.974849    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs podName:b425f704-e370-447b-a2f4-23d81057111d nodeName:}" failed. No retries permitted until 2026-09-17 18:41:19.974828086 +0000 UTC m=+131.853796828 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs") pod "network-metrics-daemon-7m9j8" (UID: "b425f704-e370-447b-a2f4-23d81057111d") : secret "metrics-daemon-secret" not found
Sep 17 18:40:15.977199 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.977170    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddd560fe-7729-4b3b-a307-d896e48b0baa-original-pull-secret\") pod \"global-pull-secret-syncer-9sj77\" (UID: \"ddd560fe-7729-4b3b-a307-d896e48b0baa\") " pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:40:15.997467 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:15.997430    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:16.065082 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:16.064985    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:40:16.065255 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:16.065164    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:40:16.065255 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:16.065239    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls podName:c65d126d-844f-4c22-b0c8-3747a48aa285 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:48.065219387 +0000 UTC m=+99.944188134 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls") pod "dns-default-9r5ml" (UID: "c65d126d-844f-4c22-b0c8-3747a48aa285") : secret "dns-default-metrics-tls" not found
Sep 17 18:40:16.065362 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:16.065256    2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:40:16.065362 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:16.065270    2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5894f4d576-l4h7z: secret "image-registry-tls" not found
Sep 17 18:40:16.065362 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:16.065170    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:40:16.065362 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:16.065316    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls podName:5b9d9afe-fb1a-4eec-9e77-9e9675ea2421 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:48.065301301 +0000 UTC m=+99.944270061 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls") pod "image-registry-5894f4d576-l4h7z" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421") : secret "image-registry-tls" not found
Sep 17 18:40:16.071998 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:16.071978    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sj77"
Sep 17 18:40:16.152097 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:16.151640    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:16.167148 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:16.166994    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qws\" (UniqueName: \"kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws\") pod \"network-check-target-7frk4\" (UID: \"3821fdb9-ccfa-42c2-b476-9c624935e87e\") " pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:40:16.170396 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:16.170375    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qws\" (UniqueName: \"kubernetes.io/projected/3821fdb9-ccfa-42c2-b476-9c624935e87e-kube-api-access-d5qws\") pod \"network-check-target-7frk4\" (UID: \"3821fdb9-ccfa-42c2-b476-9c624935e87e\") " pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:40:16.261694 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:16.261662    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:16.373557 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:16.373529    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9sj77"]
Sep 17 18:40:16.376621 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:16.376586    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd560fe_7729_4b3b_a307_d896e48b0baa.slice/crio-3f455ea2066b6a018754094df533732f96e915c8bfe75f137ec7222329b7c14f WatchSource:0}: Error finding container 3f455ea2066b6a018754094df533732f96e915c8bfe75f137ec7222329b7c14f: Status 404 returned error can't find the container with id 3f455ea2066b6a018754094df533732f96e915c8bfe75f137ec7222329b7c14f
Sep 17 18:40:16.466574 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:16.466544    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5jd5p\""
Sep 17 18:40:16.473820 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:16.473799    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:40:16.577366 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:16.577323    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9sj77" event={"ID":"ddd560fe-7729-4b3b-a307-d896e48b0baa","Type":"ContainerStarted","Data":"3f455ea2066b6a018754094df533732f96e915c8bfe75f137ec7222329b7c14f"}
Sep 17 18:40:16.750584 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:16.750483    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:16.759124 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:16.759013    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7frk4"]
Sep 17 18:40:16.761093 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:16.761070    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3821fdb9_ccfa_42c2_b476_9c624935e87e.slice/crio-6df2d0bbb3e13e7ccc6779bdcabaf2f29a382e745d93de5edfe12d59c2221804 WatchSource:0}: Error finding container 6df2d0bbb3e13e7ccc6779bdcabaf2f29a382e745d93de5edfe12d59c2221804: Status 404 returned error can't find the container with id 6df2d0bbb3e13e7ccc6779bdcabaf2f29a382e745d93de5edfe12d59c2221804
Sep 17 18:40:17.075641 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:17.075545    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-872nz\" (UID: \"01a4765a-fe1b-40b2-a611-5e815c42cc84\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz"
Sep 17 18:40:17.075813 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:17.075718    2587 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:40:17.075813 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:17.075796    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert podName:01a4765a-fe1b-40b2-a611-5e815c42cc84 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:25.075772521 +0000 UTC m=+76.954741271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-872nz" (UID: "01a4765a-fe1b-40b2-a611-5e815c42cc84") : secret "networking-console-plugin-cert" not found
Sep 17 18:40:17.191294 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:17.191223    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:17.590270 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:17.590224    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7frk4" event={"ID":"3821fdb9-ccfa-42c2-b476-9c624935e87e","Type":"ContainerStarted","Data":"cca06626763555319ef1a03719f11960c1f62dd5b5a2db97a5b89d9e3ce0cee6"}
Sep 17 18:40:17.590708 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:17.590322    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7frk4" event={"ID":"3821fdb9-ccfa-42c2-b476-9c624935e87e","Type":"ContainerStarted","Data":"6df2d0bbb3e13e7ccc6779bdcabaf2f29a382e745d93de5edfe12d59c2221804"}
Sep 17 18:40:17.590708 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:17.590358    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:40:17.609575 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:17.609516    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7frk4" podStartSLOduration=67.609497856 podStartE2EDuration="1m7.609497856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:17.608777623 +0000 UTC m=+69.487746382" watchObservedRunningTime="2026-09-17 18:40:17.609497856 +0000 UTC m=+69.488466613"
Sep 17 18:40:17.767340 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:17.767303    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:17.965161 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:17.965061    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:18.147449 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:18.147414    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:18.912400 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:18.912363    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:19.031762 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:19.031730    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:19.140583 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:19.140532    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:19.290859 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:19.290820    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:19.546425 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:19.546335    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:19.701204 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:19.701168    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:19.701391 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:19.701230    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:19.701391 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:19.701320    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:40:19.701391 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:19.701391    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm podName:857bafdf-f63a-4616-bb74-dfefd8f63fdf nodeName:}" failed. No retries permitted until 2026-09-17 18:40:27.701369964 +0000 UTC m=+79.580338713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ph9zh" (UID: "857bafdf-f63a-4616-bb74-dfefd8f63fdf") : configmap "kube-rbac-proxy" not found
Sep 17 18:40:19.701562 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:19.701391    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:19.701562 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:19.701447    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls podName:857bafdf-f63a-4616-bb74-dfefd8f63fdf nodeName:}" failed. No retries permitted until 2026-09-17 18:40:27.701431056 +0000 UTC m=+79.580399806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ph9zh" (UID: "857bafdf-f63a-4616-bb74-dfefd8f63fdf") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:19.751548 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:19.751424    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:19.870652 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:19.870569    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:19.973924 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:19.973889    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:20.307113 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:20.307077    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:20.307317 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:20.307157    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:20.307317 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:20.307179    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:20.307317 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:20.307264    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle podName:ad67d410-cd73-46d3-81b8-d409d33ca41c nodeName:}" failed. No retries permitted until 2026-09-17 18:40:36.307240737 +0000 UTC m=+88.186209485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle") pod "router-default-7646b475dd-4rb55" (UID: "ad67d410-cd73-46d3-81b8-d409d33ca41c") : configmap references non-existent config key: service-ca.crt
Sep 17 18:40:20.307317 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:20.307283    2587 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:40:20.307548 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:20.307332    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls podName:1ca5000b-7618-4602-8388-1c272aa30ae8 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:36.307317511 +0000 UTC m=+88.186286248 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qcr7l" (UID: "1ca5000b-7618-4602-8388-1c272aa30ae8") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:40:20.309653 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:20.309629    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad67d410-cd73-46d3-81b8-d409d33ca41c-metrics-certs\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:20.374157 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:20.374119    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:20.409056 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:20.408386    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77q28\" (UID: \"b81db7d2-6a61-4665-8e55-f22e29e5e30f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"
Sep 17 18:40:20.412114 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:20.412085    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b81db7d2-6a61-4665-8e55-f22e29e5e30f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77q28\" (UID: \"b81db7d2-6a61-4665-8e55-f22e29e5e30f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"
Sep 17 18:40:20.506135 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:20.506102    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9vvmx\""
Sep 17 18:40:20.511645 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:20.511624    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"
Sep 17 18:40:20.603199 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:20.603155    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9sj77" event={"ID":"ddd560fe-7729-4b3b-a307-d896e48b0baa","Type":"ContainerStarted","Data":"fc4fcabe90ede349c6b8f6825bf8ceea5a110317af0f18f668507d9d8182279e"}
Sep 17 18:40:20.623555 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:20.623508    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9sj77" podStartSLOduration=66.815187261 podStartE2EDuration="1m10.623491589s" totalImagesPullingTime="3.808304328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:20.621624348 +0000 UTC m=+72.500593105" watchObservedRunningTime="2026-09-17 18:40:20.623491589 +0000 UTC m=+72.502460346"
Sep 17 18:40:20.799439 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:20.799392    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28"]
Sep 17 18:40:21.134500 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:21.134405    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:21.336873 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:21.336839    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:21.608367 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:21.608250    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28" event={"ID":"b81db7d2-6a61-4665-8e55-f22e29e5e30f","Type":"ContainerStarted","Data":"46d0c4e3e7294a3634589cbe37e65d64cf2c252458e3a2c3c8baa9ee3e486085"}
Sep 17 18:40:21.740993 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:21.740957    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:21.822619 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:21.822566    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:21.939928 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:21.939843    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:22.330117 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:22.330092    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:22.613357 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:22.613322    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28" event={"ID":"b81db7d2-6a61-4665-8e55-f22e29e5e30f","Type":"ContainerStarted","Data":"ef3076bd938c5d013e4683c2f7db75c0b07d8a196942214fdc424ad75f90ae5f"}
Sep 17 18:40:22.613527 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:22.613378    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28" event={"ID":"b81db7d2-6a61-4665-8e55-f22e29e5e30f","Type":"ContainerStarted","Data":"449cc44c176815be3ec481c5045738047b3e7e8cadd3b431a4acfd12e1e491c7"}
Sep 17 18:40:22.632628 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:22.632576    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77q28" podStartSLOduration=17.23755214 podStartE2EDuration="18.632560437s" totalImagesPullingTime="1.395008297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:22.631909212 +0000 UTC m=+74.510877969" watchObservedRunningTime="2026-09-17 18:40:22.632560437 +0000 UTC m=+74.511529227"
Sep 17 18:40:23.129590 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:23.129557    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:23.200264 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:23.200233    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:23.335284 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:23.335251    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:23.558693 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:23.558633    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:23.736822 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:23.736783    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:24.728749 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:24.728716    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:24.829727 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:24.829680    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:25.123637 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:25.123600    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:25.153672 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:25.153638    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-872nz\" (UID: \"01a4765a-fe1b-40b2-a611-5e815c42cc84\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz"
Sep 17 18:40:25.156250 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:25.156227    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/01a4765a-fe1b-40b2-a611-5e815c42cc84-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-872nz\" (UID: \"01a4765a-fe1b-40b2-a611-5e815c42cc84\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz"
Sep 17 18:40:25.212210 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:25.212180    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4bpj2\""
Sep 17 18:40:25.220421 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:25.220394    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz"
Sep 17 18:40:25.326896 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:25.326865    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:25.490064 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:25.490032    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-872nz"]
Sep 17 18:40:25.494456 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:25.494415    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a4765a_fe1b_40b2_a611_5e815c42cc84.slice/crio-562be0b4cd335c3dd188d4aaa9d701a1da1b6e87e37ffa70659ab54bee181e07 WatchSource:0}: Error finding container 562be0b4cd335c3dd188d4aaa9d701a1da1b6e87e37ffa70659ab54bee181e07: Status 404 returned error can't find the container with id 562be0b4cd335c3dd188d4aaa9d701a1da1b6e87e37ffa70659ab54bee181e07
Sep 17 18:40:25.539928 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:25.539867    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:25.625738 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:25.625708    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz" event={"ID":"01a4765a-fe1b-40b2-a611-5e815c42cc84","Type":"ContainerStarted","Data":"562be0b4cd335c3dd188d4aaa9d701a1da1b6e87e37ffa70659ab54bee181e07"}
Sep 17 18:40:25.725254 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:25.725222    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:26.133544 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:26.133447    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:26.252742 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:26.252713    2587 scope.go:117] "RemoveContainer" containerID="b8c84218bc1ed393e39031a8118406fdb89d0a4840457a871800f954e4829ed6"
Sep 17 18:40:26.631705 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:26.631675    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 18:40:26.631885 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:26.631769    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml" event={"ID":"1e4b5a1a-b85d-4ff5-9741-99678d16aad1","Type":"ContainerStarted","Data":"03e0851b9246db5e31251efda4bcb8368fe8a1e98744fdc52caa08ce1614b818"}
Sep 17 18:40:26.632211 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:26.632187    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:26.633581 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:26.633532    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz" event={"ID":"01a4765a-fe1b-40b2-a611-5e815c42cc84","Type":"ContainerStarted","Data":"13ab7697f21bc15a1325d114c5b35fb7da74e3f6e9d7d8b21bfdc07fac2bbf67"}
Sep 17 18:40:26.651608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:26.651564    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml" podStartSLOduration=17.518356503 podStartE2EDuration="22.651549193s" totalImagesPullingTime="5.13319269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:26.650645578 +0000 UTC m=+78.529614335" watchObservedRunningTime="2026-09-17 18:40:26.651549193 +0000 UTC m=+78.530517950"
Sep 17 18:40:26.666793 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:26.666745    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-872nz" podStartSLOduration=16.645841337 podStartE2EDuration="17.666730919s" totalImagesPullingTime="1.020889582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:26.666545333 +0000 UTC m=+78.545514092" watchObservedRunningTime="2026-09-17 18:40:26.666730919 +0000 UTC m=+78.545699676"
Sep 17 18:40:26.737460 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:26.737423    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:27.554510 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:27.554476    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-fm5ml"
Sep 17 18:40:27.778916 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:27.778882    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:27.779133 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:27.778930    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:27.779523 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:27.779499    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/857bafdf-f63a-4616-bb74-dfefd8f63fdf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:27.781552 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:27.781531    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/857bafdf-f63a-4616-bb74-dfefd8f63fdf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ph9zh\" (UID: \"857bafdf-f63a-4616-bb74-dfefd8f63fdf\") " pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:27.796436 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:27.796397    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:27.830594 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:27.830505    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ph9zh"
Sep 17 18:40:27.896655 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:27.896628    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:28.047437 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:28.047393    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:28.114998 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:28.114859    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ph9zh"]
Sep 17 18:40:28.117529 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:28.117506    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857bafdf_f63a_4616_bb74_dfefd8f63fdf.slice/crio-63f91144f7d24e6d5f4f0a09ff4a5bc9d211d5670ad4a3980498a0ee22c30aa3 WatchSource:0}: Error finding container 63f91144f7d24e6d5f4f0a09ff4a5bc9d211d5670ad4a3980498a0ee22c30aa3: Status 404 returned error can't find the container with id 63f91144f7d24e6d5f4f0a09ff4a5bc9d211d5670ad4a3980498a0ee22c30aa3
Sep 17 18:40:28.497839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:28.497802    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:28.642624 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:28.642572    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ph9zh" event={"ID":"857bafdf-f63a-4616-bb74-dfefd8f63fdf","Type":"ContainerStarted","Data":"aeca2b6b4a9e60efee72ca274e529602b1ab66baf70a86361dbce4ec0563d076"}
Sep 17 18:40:28.643138 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:28.642644    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ph9zh" event={"ID":"857bafdf-f63a-4616-bb74-dfefd8f63fdf","Type":"ContainerStarted","Data":"63f91144f7d24e6d5f4f0a09ff4a5bc9d211d5670ad4a3980498a0ee22c30aa3"}
Sep 17 18:40:28.658284 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:28.658248    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:28.748681 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:28.748596    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:29.245560 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:29.245514    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:29.593633 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:29.593513    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:29.649200 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:29.649166    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ph9zh" event={"ID":"857bafdf-f63a-4616-bb74-dfefd8f63fdf","Type":"ContainerStarted","Data":"35efeb4e32547f23473e22b21ad249e4ffe8e69649940b096f15553aa73f34b1"}
Sep 17 18:40:29.692610 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:29.692574    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:29.871910 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:29.871832    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:30.259113 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:30.259079    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:30.361237 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:30.361204    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:30.654488 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:30.654449    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ph9zh" event={"ID":"857bafdf-f63a-4616-bb74-dfefd8f63fdf","Type":"ContainerStarted","Data":"50b19cd61e4edebf174709ac05e1d1340f886e218cc50347ca8bc80dc7971c21"}
Sep 17 18:40:30.680008 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:30.679957    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ph9zh" podStartSLOduration=17.793959631 podStartE2EDuration="19.67992429s" totalImagesPullingTime="1.885964659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:40:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:30.678521927 +0000 UTC m=+82.557490683" watchObservedRunningTime="2026-09-17 18:40:30.67992429 +0000 UTC m=+82.558893080"
Sep 17 18:40:34.652973 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.652927    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln"]
Sep 17 18:40:34.655549 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.655530    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln"
Sep 17 18:40:34.668540 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:34.668513    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"managed-serviceaccount-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-140-194.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-140-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" type="*v1.Secret"
Sep 17 18:40:34.672453 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.672431    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"]
Sep 17 18:40:34.673386 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:34.673362    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"open-cluster-management-image-pull-credentials\" is forbidden: User \"system:node:ip-10-0-140-194.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-140-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret"
Sep 17 18:40:34.673576 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:34.673559    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-194.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-140-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 17 18:40:34.674546 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.674528    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.674644 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:34.674611    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"managed-serviceaccount-dockercfg-kxjpk\" is forbidden: User \"system:node:ip-10-0-140-194.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-140-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-kxjpk\"" type="*v1.Secret"
Sep 17 18:40:34.675243 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:34.675187    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln\" is forbidden: User \"system:node:ip-10-0-140-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-140-194.ec2.internal' and this object" podUID="4ff5af59-daa6-47d9-8a6b-367ce63b2060" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln"
Sep 17 18:40:34.680302 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.680283    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 18:40:34.683715 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.683697    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 18:40:34.684117 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.683987    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 18:40:34.688124 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.688103    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 18:40:34.688578 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.688563    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 18:40:34.703218 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.703183    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln"]
Sep 17 18:40:34.711148 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.711122    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"]
Sep 17 18:40:34.713701 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.713673    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"
Sep 17 18:40:34.715446 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.715402    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"]
Sep 17 18:40:34.717136 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.717115    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 18:40:34.744766 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.744741    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-4cd7g"]
Sep 17 18:40:34.746917 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.746897    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-4cd7g"
Sep 17 18:40:34.757181 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.757152    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"]
Sep 17 18:40:34.768363 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.768326    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 18:40:34.768511 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.768327    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fvhp6\""
Sep 17 18:40:34.768511 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.768327    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 18:40:34.841260 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.841220    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxtzn\" (UniqueName: \"kubernetes.io/projected/c3d93259-1f5a-44c1-815f-ea577fbdabb7-kube-api-access-pxtzn\") pod \"downloads-6594f4c75-4cd7g\" (UID: \"c3d93259-1f5a-44c1-815f-ea577fbdabb7\") " pod="openshift-console/downloads-6594f4c75-4cd7g"
Sep 17 18:40:34.841411 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.841292    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/66ff5223-4d0b-49ab-8dc8-add699a25312-klusterlet-config\") pod \"klusterlet-addon-workmgr-858dfdf88f-x5rp5\" (UID: \"66ff5223-4d0b-49ab-8dc8-add699a25312\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"
Sep 17 18:40:34.841488 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.841427    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66ff5223-4d0b-49ab-8dc8-add699a25312-tmp\") pod \"klusterlet-addon-workmgr-858dfdf88f-x5rp5\" (UID: \"66ff5223-4d0b-49ab-8dc8-add699a25312\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"
Sep 17 18:40:34.841488 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.841457    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/105a6897-8010-47ac-8eb5-ec5417d1e263-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.841488 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.841477    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/105a6897-8010-47ac-8eb5-ec5417d1e263-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.841619 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.841505    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/105a6897-8010-47ac-8eb5-ec5417d1e263-ca\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.841667 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.841620    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/105a6897-8010-47ac-8eb5-ec5417d1e263-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.841724 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.841682    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fj95\" (UniqueName: \"kubernetes.io/projected/105a6897-8010-47ac-8eb5-ec5417d1e263-kube-api-access-5fj95\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.841724 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.841713    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzt4\" (UniqueName: \"kubernetes.io/projected/4ff5af59-daa6-47d9-8a6b-367ce63b2060-kube-api-access-qkzt4\") pod \"managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln\" (UID: \"4ff5af59-daa6-47d9-8a6b-367ce63b2060\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln"
Sep 17 18:40:34.841820 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.841755    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/105a6897-8010-47ac-8eb5-ec5417d1e263-hub\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.841820 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.841793    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qs4b\" (UniqueName: \"kubernetes.io/projected/66ff5223-4d0b-49ab-8dc8-add699a25312-kube-api-access-2qs4b\") pod \"klusterlet-addon-workmgr-858dfdf88f-x5rp5\" (UID: \"66ff5223-4d0b-49ab-8dc8-add699a25312\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"
Sep 17 18:40:34.841903 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.841820    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ff5af59-daa6-47d9-8a6b-367ce63b2060-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln\" (UID: \"4ff5af59-daa6-47d9-8a6b-367ce63b2060\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln"
Sep 17 18:40:34.850102 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.850071    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-4cd7g"]
Sep 17 18:40:34.942608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.942510    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66ff5223-4d0b-49ab-8dc8-add699a25312-tmp\") pod \"klusterlet-addon-workmgr-858dfdf88f-x5rp5\" (UID: \"66ff5223-4d0b-49ab-8dc8-add699a25312\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"
Sep 17 18:40:34.942608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.942575    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/105a6897-8010-47ac-8eb5-ec5417d1e263-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.942608 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.942608    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/105a6897-8010-47ac-8eb5-ec5417d1e263-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.942878 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.942650    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/105a6897-8010-47ac-8eb5-ec5417d1e263-ca\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.942878 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.942697    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/105a6897-8010-47ac-8eb5-ec5417d1e263-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.942878 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.942729    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fj95\" (UniqueName: \"kubernetes.io/projected/105a6897-8010-47ac-8eb5-ec5417d1e263-kube-api-access-5fj95\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.942878 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.942753    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzt4\" (UniqueName: \"kubernetes.io/projected/4ff5af59-daa6-47d9-8a6b-367ce63b2060-kube-api-access-qkzt4\") pod \"managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln\" (UID: \"4ff5af59-daa6-47d9-8a6b-367ce63b2060\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln"
Sep 17 18:40:34.942878 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.942789    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/105a6897-8010-47ac-8eb5-ec5417d1e263-hub\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.942878 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.942816    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qs4b\" (UniqueName: \"kubernetes.io/projected/66ff5223-4d0b-49ab-8dc8-add699a25312-kube-api-access-2qs4b\") pod \"klusterlet-addon-workmgr-858dfdf88f-x5rp5\" (UID: \"66ff5223-4d0b-49ab-8dc8-add699a25312\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"
Sep 17 18:40:34.942878 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.942842    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ff5af59-daa6-47d9-8a6b-367ce63b2060-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln\" (UID: \"4ff5af59-daa6-47d9-8a6b-367ce63b2060\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln"
Sep 17 18:40:34.942878 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.942869    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxtzn\" (UniqueName: \"kubernetes.io/projected/c3d93259-1f5a-44c1-815f-ea577fbdabb7-kube-api-access-pxtzn\") pod \"downloads-6594f4c75-4cd7g\" (UID: \"c3d93259-1f5a-44c1-815f-ea577fbdabb7\") " pod="openshift-console/downloads-6594f4c75-4cd7g"
Sep 17 18:40:34.943238 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.942900    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/66ff5223-4d0b-49ab-8dc8-add699a25312-klusterlet-config\") pod \"klusterlet-addon-workmgr-858dfdf88f-x5rp5\" (UID: \"66ff5223-4d0b-49ab-8dc8-add699a25312\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"
Sep 17 18:40:34.943989 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.943914    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/105a6897-8010-47ac-8eb5-ec5417d1e263-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.944223 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.944066    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66ff5223-4d0b-49ab-8dc8-add699a25312-tmp\") pod \"klusterlet-addon-workmgr-858dfdf88f-x5rp5\" (UID: \"66ff5223-4d0b-49ab-8dc8-add699a25312\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"
Sep 17 18:40:34.946124 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.946098    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/105a6897-8010-47ac-8eb5-ec5417d1e263-ca\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.946283 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.946131    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/105a6897-8010-47ac-8eb5-ec5417d1e263-hub\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.946424 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.946404    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/66ff5223-4d0b-49ab-8dc8-add699a25312-klusterlet-config\") pod \"klusterlet-addon-workmgr-858dfdf88f-x5rp5\" (UID: \"66ff5223-4d0b-49ab-8dc8-add699a25312\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"
Sep 17 18:40:34.946558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.946541    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/105a6897-8010-47ac-8eb5-ec5417d1e263-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.946840 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.946820    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/105a6897-8010-47ac-8eb5-ec5417d1e263-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:34.956134 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.956104    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:34.983804 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:34.983772    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxtzn\" (UniqueName: \"kubernetes.io/projected/c3d93259-1f5a-44c1-815f-ea577fbdabb7-kube-api-access-pxtzn\") pod \"downloads-6594f4c75-4cd7g\" (UID: \"c3d93259-1f5a-44c1-815f-ea577fbdabb7\") " pod="openshift-console/downloads-6594f4c75-4cd7g"
Sep 17 18:40:35.056335 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:35.056295    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-4cd7g"
Sep 17 18:40:35.114530 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:35.114482    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:35.234465 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:35.234439    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:35.346876 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:35.346839    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-4cd7g"]
Sep 17 18:40:35.350054 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:35.350022    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d93259_1f5a_44c1_815f_ea577fbdabb7.slice/crio-246dccd4402f0887556f26e0d8c832f88bd1e17edce78077e0c40504aad668d3 WatchSource:0}: Error finding container 246dccd4402f0887556f26e0d8c832f88bd1e17edce78077e0c40504aad668d3: Status 404 returned error can't find the container with id 246dccd4402f0887556f26e0d8c832f88bd1e17edce78077e0c40504aad668d3
Sep 17 18:40:35.379605 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:35.379572    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:35.662509 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:35.662427    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 18:40:35.666926 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:35.666901    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ff5af59-daa6-47d9-8a6b-367ce63b2060-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln\" (UID: \"4ff5af59-daa6-47d9-8a6b-367ce63b2060\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln"
Sep 17 18:40:35.672193 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:35.672142    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-4cd7g" event={"ID":"c3d93259-1f5a-44c1-815f-ea577fbdabb7","Type":"ContainerStarted","Data":"246dccd4402f0887556f26e0d8c832f88bd1e17edce78077e0c40504aad668d3"}
Sep 17 18:40:35.982286 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:35.982199    2587 projected.go:291] Couldn't get configMap open-cluster-management-agent-addon/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 17 18:40:35.982286 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:35.982224    2587 projected.go:291] Couldn't get configMap open-cluster-management-agent-addon/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 17 18:40:35.982286 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:35.982255    2587 projected.go:196] Error preparing data for projected volume kube-api-access-qkzt4 for pod open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln: failed to sync configmap cache: timed out waiting for the condition
Sep 17 18:40:35.982286 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:35.982268    2587 projected.go:196] Error preparing data for projected volume kube-api-access-5fj95 for pod open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w: failed to sync configmap cache: timed out waiting for the condition
Sep 17 18:40:35.982642 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:35.982214    2587 projected.go:291] Couldn't get configMap open-cluster-management-agent-addon/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 17 18:40:35.982642 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:35.982337    2587 projected.go:196] Error preparing data for projected volume kube-api-access-2qs4b for pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5: failed to sync configmap cache: timed out waiting for the condition
Sep 17 18:40:35.982642 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:35.982335    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/105a6897-8010-47ac-8eb5-ec5417d1e263-kube-api-access-5fj95 podName:105a6897-8010-47ac-8eb5-ec5417d1e263 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:36.482313491 +0000 UTC m=+88.361282229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5fj95" (UniqueName: "kubernetes.io/projected/105a6897-8010-47ac-8eb5-ec5417d1e263-kube-api-access-5fj95") pod "cluster-proxy-proxy-agent-78d9797c77-w4d4w" (UID: "105a6897-8010-47ac-8eb5-ec5417d1e263") : failed to sync configmap cache: timed out waiting for the condition
Sep 17 18:40:35.982642 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:35.982387    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ff5af59-daa6-47d9-8a6b-367ce63b2060-kube-api-access-qkzt4 podName:4ff5af59-daa6-47d9-8a6b-367ce63b2060 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:36.482369461 +0000 UTC m=+88.361338203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qkzt4" (UniqueName: "kubernetes.io/projected/4ff5af59-daa6-47d9-8a6b-367ce63b2060-kube-api-access-qkzt4") pod "managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln" (UID: "4ff5af59-daa6-47d9-8a6b-367ce63b2060") : failed to sync configmap cache: timed out waiting for the condition
Sep 17 18:40:35.982642 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:35.982408    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66ff5223-4d0b-49ab-8dc8-add699a25312-kube-api-access-2qs4b podName:66ff5223-4d0b-49ab-8dc8-add699a25312 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:36.48240098 +0000 UTC m=+88.361369714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2qs4b" (UniqueName: "kubernetes.io/projected/66ff5223-4d0b-49ab-8dc8-add699a25312-kube-api-access-2qs4b") pod "klusterlet-addon-workmgr-858dfdf88f-x5rp5" (UID: "66ff5223-4d0b-49ab-8dc8-add699a25312") : failed to sync configmap cache: timed out waiting for the condition
Sep 17 18:40:35.994556 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:35.994524    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 18:40:36.033816 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.033781    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 18:40:36.076548 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.076511    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-kxjpk\""
Sep 17 18:40:36.356375 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.356339    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:36.356548 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.356445    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:36.357250 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.357172    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad67d410-cd73-46d3-81b8-d409d33ca41c-service-ca-bundle\") pod \"router-default-7646b475dd-4rb55\" (UID: \"ad67d410-cd73-46d3-81b8-d409d33ca41c\") " pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:36.359306 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.359279    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ca5000b-7618-4602-8388-1c272aa30ae8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qcr7l\" (UID: \"1ca5000b-7618-4602-8388-1c272aa30ae8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:36.557874 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.557837    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qs4b\" (UniqueName: \"kubernetes.io/projected/66ff5223-4d0b-49ab-8dc8-add699a25312-kube-api-access-2qs4b\") pod \"klusterlet-addon-workmgr-858dfdf88f-x5rp5\" (UID: \"66ff5223-4d0b-49ab-8dc8-add699a25312\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"
Sep 17 18:40:36.558087 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.558000    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fj95\" (UniqueName: \"kubernetes.io/projected/105a6897-8010-47ac-8eb5-ec5417d1e263-kube-api-access-5fj95\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:36.558087 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.558025    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzt4\" (UniqueName: \"kubernetes.io/projected/4ff5af59-daa6-47d9-8a6b-367ce63b2060-kube-api-access-qkzt4\") pod \"managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln\" (UID: \"4ff5af59-daa6-47d9-8a6b-367ce63b2060\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln"
Sep 17 18:40:36.561084 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.561056    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fj95\" (UniqueName: \"kubernetes.io/projected/105a6897-8010-47ac-8eb5-ec5417d1e263-kube-api-access-5fj95\") pod \"cluster-proxy-proxy-agent-78d9797c77-w4d4w\" (UID: \"105a6897-8010-47ac-8eb5-ec5417d1e263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:36.561212 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.561109    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qs4b\" (UniqueName: \"kubernetes.io/projected/66ff5223-4d0b-49ab-8dc8-add699a25312-kube-api-access-2qs4b\") pod \"klusterlet-addon-workmgr-858dfdf88f-x5rp5\" (UID: \"66ff5223-4d0b-49ab-8dc8-add699a25312\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"
Sep 17 18:40:36.561212 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.561166    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzt4\" (UniqueName: \"kubernetes.io/projected/4ff5af59-daa6-47d9-8a6b-367ce63b2060-kube-api-access-qkzt4\") pod \"managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln\" (UID: \"4ff5af59-daa6-47d9-8a6b-367ce63b2060\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln"
Sep 17 18:40:36.567834 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.567807    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7shh9\""
Sep 17 18:40:36.575246 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.575224    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"
Sep 17 18:40:36.590765 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.590739    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tczk5\""
Sep 17 18:40:36.597640 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.597611    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:36.775790 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.775760    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln"
Sep 17 18:40:36.783338 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.783317    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"
Sep 17 18:40:36.843036 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.840388    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"
Sep 17 18:40:36.892109 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.892080    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l"]
Sep 17 18:40:36.894739 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:36.894706    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca5000b_7618_4602_8388_1c272aa30ae8.slice/crio-4c3807e39f53f18522021d493a8e03fb121415c4849afc4e2786d5c7673e677b WatchSource:0}: Error finding container 4c3807e39f53f18522021d493a8e03fb121415c4849afc4e2786d5c7673e677b: Status 404 returned error can't find the container with id 4c3807e39f53f18522021d493a8e03fb121415c4849afc4e2786d5c7673e677b
Sep 17 18:40:36.943216 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:36.943192    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7646b475dd-4rb55"]
Sep 17 18:40:36.945229 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:36.945201    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad67d410_cd73_46d3_81b8_d409d33ca41c.slice/crio-9cb7b735c2eddbc02b3a47cf6cd0746afe24a025865abc6aac7d9f785cc9c951 WatchSource:0}: Error finding container 9cb7b735c2eddbc02b3a47cf6cd0746afe24a025865abc6aac7d9f785cc9c951: Status 404 returned error can't find the container with id 9cb7b735c2eddbc02b3a47cf6cd0746afe24a025865abc6aac7d9f785cc9c951
Sep 17 18:40:37.077541 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:37.077499    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln"]
Sep 17 18:40:37.084405 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:37.084346    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w"]
Sep 17 18:40:37.156846 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:37.155184    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"]
Sep 17 18:40:37.163102 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:37.163070    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ff5223_4d0b_49ab_8dc8_add699a25312.slice/crio-1cb1c22e5930000399f680f0736e92a12ff6aafc4ea71576c5109aed23dcee21 WatchSource:0}: Error finding container 1cb1c22e5930000399f680f0736e92a12ff6aafc4ea71576c5109aed23dcee21: Status 404 returned error can't find the container with id 1cb1c22e5930000399f680f0736e92a12ff6aafc4ea71576c5109aed23dcee21
Sep 17 18:40:37.685963 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:37.685879    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln" event={"ID":"4ff5af59-daa6-47d9-8a6b-367ce63b2060","Type":"ContainerStarted","Data":"9a7c4efeae124caffa37dbd6bbc26602f7735758e781c774fa8ec4c57f396509"}
Sep 17 18:40:37.688398 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:37.688368    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7646b475dd-4rb55" event={"ID":"ad67d410-cd73-46d3-81b8-d409d33ca41c","Type":"ContainerStarted","Data":"2240ca313b04f14137ab75e1227294cb1f4cad898bb8a29dbfada04998833d30"}
Sep 17 18:40:37.688531 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:37.688422    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7646b475dd-4rb55" event={"ID":"ad67d410-cd73-46d3-81b8-d409d33ca41c","Type":"ContainerStarted","Data":"9cb7b735c2eddbc02b3a47cf6cd0746afe24a025865abc6aac7d9f785cc9c951"}
Sep 17 18:40:37.689970 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:37.689926    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5" event={"ID":"66ff5223-4d0b-49ab-8dc8-add699a25312","Type":"ContainerStarted","Data":"1cb1c22e5930000399f680f0736e92a12ff6aafc4ea71576c5109aed23dcee21"}
Sep 17 18:40:37.691603 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:37.691580    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l" event={"ID":"1ca5000b-7618-4602-8388-1c272aa30ae8","Type":"ContainerStarted","Data":"4c3807e39f53f18522021d493a8e03fb121415c4849afc4e2786d5c7673e677b"}
Sep 17 18:40:37.693837 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:37.693812    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w" event={"ID":"105a6897-8010-47ac-8eb5-ec5417d1e263","Type":"ContainerStarted","Data":"fe1f041e22b8e03d8a8da52ba5de96c6a71a9ffe200cbea2720391eaf9f5c57f"}
Sep 17 18:40:37.736156 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:37.736094    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7646b475dd-4rb55" podStartSLOduration=33.73607448 podStartE2EDuration="33.73607448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:37.734318021 +0000 UTC m=+89.613286790" watchObservedRunningTime="2026-09-17 18:40:37.73607448 +0000 UTC m=+89.615043239"
Sep 17 18:40:38.189812 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:38.189742    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:38.291687 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:38.291575    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:38.598991 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:38.598814    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:38.601951 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:38.601909    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:38.700594 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:38.700565    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:38.701784 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:38.701763    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7646b475dd-4rb55"
Sep 17 18:40:40.802444 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:40.802414    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:40.881466 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:40.881425    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:41.466089 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:41.466050    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:41.550925 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:41.550891    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:42.661310 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:42.661252    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:42.806436 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:42.806371    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:44.447730 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:44.447638    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:44.513811 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:44.513736    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:45.074815 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:45.074703    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:45.190295 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:45.190260    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:46.188428 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.188386    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:46.285459 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.285357    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:46.393704 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.393615    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:46.640298 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.640207    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:46.740375 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.740322    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln" event={"ID":"4ff5af59-daa6-47d9-8a6b-367ce63b2060","Type":"ContainerStarted","Data":"e5deb793b2c08af0ef094c83d4287a6f803e90fe300147c6933ff9de76e72337"}
Sep 17 18:40:46.742103 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.742071    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5" event={"ID":"66ff5223-4d0b-49ab-8dc8-add699a25312","Type":"ContainerStarted","Data":"4e344a75fc55f2747739cac5a1389f1221f09489376ce662c8257a833dd072bf"}
Sep 17 18:40:46.742242 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.742130    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"
Sep 17 18:40:46.743859 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.743833    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l" event={"ID":"1ca5000b-7618-4602-8388-1c272aa30ae8","Type":"ContainerStarted","Data":"5a18aed271c2a132edf082aba34320cbcc89c9cf19ad0184dcb37f8c9cfdbef7"}
Sep 17 18:40:46.744316 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.744296    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5"
Sep 17 18:40:46.746187 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.746167    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w" event={"ID":"105a6897-8010-47ac-8eb5-ec5417d1e263","Type":"ContainerStarted","Data":"bf98afa1eb8d7671c3b97f26b808ee808aad29d2e1a01030c01b5062bb88ba6f"}
Sep 17 18:40:46.746270 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.746197    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w" event={"ID":"105a6897-8010-47ac-8eb5-ec5417d1e263","Type":"ContainerStarted","Data":"c35cf677f970c809856c702388995f1b5a07462e0b6221fa8552b5d5a29c8567"}
Sep 17 18:40:46.746270 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.746208    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w" event={"ID":"105a6897-8010-47ac-8eb5-ec5417d1e263","Type":"ContainerStarted","Data":"67b5d95da101a948ca534b4b0da16db39e4a18732bb28ef3d64ec3e99dcc2c84"}
Sep 17 18:40:46.758293 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.758245    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c47f6cc46-wr8ln" podStartSLOduration=4.085711515 podStartE2EDuration="12.758228423s" totalImagesPullingTime="8.672516908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:46.756979513 +0000 UTC m=+98.635948272" watchObservedRunningTime="2026-09-17 18:40:46.758228423 +0000 UTC m=+98.637197180"
Sep 17 18:40:46.779473 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.779113    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d9797c77-w4d4w" podStartSLOduration=4.0876943709999995 podStartE2EDuration="12.779096955s" totalImagesPullingTime="8.691402584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:46.778325653 +0000 UTC m=+98.657294414" watchObservedRunningTime="2026-09-17 18:40:46.779096955 +0000 UTC m=+98.658065711"
Sep 17 18:40:46.797670 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.797601    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-858dfdf88f-x5rp5" podStartSLOduration=4.17408133 podStartE2EDuration="12.797585723s" totalImagesPullingTime="8.623504393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:46.79564108 +0000 UTC m=+98.674609837" watchObservedRunningTime="2026-09-17 18:40:46.797585723 +0000 UTC m=+98.676554480"
Sep 17 18:40:46.815253 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:46.815194    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qcr7l" podStartSLOduration=33.952135386 podStartE2EDuration="42.815124125s" totalImagesPullingTime="8.862988739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:46.814644478 +0000 UTC m=+98.693613235" watchObservedRunningTime="2026-09-17 18:40:46.815124125 +0000 UTC m=+98.694092882"
Sep 17 18:40:47.305645 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:47.305596    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:47.393083 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:47.393042    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:48.077552 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:48.077509    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:40:48.077852 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:48.077650    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:40:48.080407 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:48.080381    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c65d126d-844f-4c22-b0c8-3747a48aa285-metrics-tls\") pod \"dns-default-9r5ml\" (UID: \"c65d126d-844f-4c22-b0c8-3747a48aa285\") " pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:40:48.080668 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:48.080640    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls\") pod \"image-registry-5894f4d576-l4h7z\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") " pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:40:48.086019 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:48.085995    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-spfxb\""
Sep 17 18:40:48.093726 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:48.093681    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:40:48.101072 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:48.101047    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gj7h2\""
Sep 17 18:40:48.108501 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:48.108473    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:40:48.265565 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:48.265527    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:48.349158 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:48.349078    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:48.597001 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:48.596968    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7frk4"
Sep 17 18:40:50.031678 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:50.031622    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:50.174258 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:50.174172    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:50.601555 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:50.601517    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:50.834247 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:50.834203    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:52.980628 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:52.980518    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:53.081609 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:53.081559    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:53.275636 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:53.275594    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:53.647277 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:53.647172    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:53.728039 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:53.727991    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:54.169057 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:54.169034    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5894f4d576-l4h7z"]
Sep 17 18:40:54.171541 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:54.171515    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9r5ml"]
Sep 17 18:40:54.205295 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:54.205225    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9d9afe_fb1a_4eec_9e77_9e9675ea2421.slice/crio-8ef48ce238ffdea8330c04f1495f76dc4d0001998a4c3287c3ab6cd1cd238259 WatchSource:0}: Error finding container 8ef48ce238ffdea8330c04f1495f76dc4d0001998a4c3287c3ab6cd1cd238259: Status 404 returned error can't find the container with id 8ef48ce238ffdea8330c04f1495f76dc4d0001998a4c3287c3ab6cd1cd238259
Sep 17 18:40:54.206211 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:54.206183    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65d126d_844f_4c22_b0c8_3747a48aa285.slice/crio-792940b820ea113ac51cc38fe68d310a5224c83d716a5f4e9756fa0d491caaf6 WatchSource:0}: Error finding container 792940b820ea113ac51cc38fe68d310a5224c83d716a5f4e9756fa0d491caaf6: Status 404 returned error can't find the container with id 792940b820ea113ac51cc38fe68d310a5224c83d716a5f4e9756fa0d491caaf6
Sep 17 18:40:54.747552 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:54.746473    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:54.791679 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:54.791621    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5894f4d576-l4h7z" event={"ID":"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421","Type":"ContainerStarted","Data":"993328358add779d983dfd0c12a4a56456d9733231b824b23b5ed5719cfb883f"}
Sep 17 18:40:54.793088 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:54.793055    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5894f4d576-l4h7z" event={"ID":"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421","Type":"ContainerStarted","Data":"8ef48ce238ffdea8330c04f1495f76dc4d0001998a4c3287c3ab6cd1cd238259"}
Sep 17 18:40:54.793270 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:54.793252    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:40:54.795112 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:54.795093    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9r5ml" event={"ID":"c65d126d-844f-4c22-b0c8-3747a48aa285","Type":"ContainerStarted","Data":"792940b820ea113ac51cc38fe68d310a5224c83d716a5f4e9756fa0d491caaf6"}
Sep 17 18:40:54.798137 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:54.797142    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-4cd7g" event={"ID":"c3d93259-1f5a-44c1-815f-ea577fbdabb7","Type":"ContainerStarted","Data":"7fe99bcf92d870d2f4715861d7ec7a3689d69128e80711d038939782da32874e"}
Sep 17 18:40:54.798137 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:54.797807    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-4cd7g"
Sep 17 18:40:54.805727 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:54.805633    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-4cd7g"
Sep 17 18:40:54.825759 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:54.825697    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5894f4d576-l4h7z" podStartSLOduration=83.82567636 podStartE2EDuration="1m23.82567636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:54.822821039 +0000 UTC m=+106.701789802" watchObservedRunningTime="2026-09-17 18:40:54.82567636 +0000 UTC m=+106.704645118"
Sep 17 18:40:54.854273 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:54.854186    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-4cd7g" podStartSLOduration=1.931205123 podStartE2EDuration="20.85416125s" totalImagesPullingTime="18.922956127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:54.851288435 +0000 UTC m=+106.730257195" watchObservedRunningTime="2026-09-17 18:40:54.85416125 +0000 UTC m=+106.733130009"
Sep 17 18:40:55.053959 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.053852    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:55.145589 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.145557    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wtwnk"]
Sep 17 18:40:55.170576 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.170543    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.177984 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.177956    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 18:40:55.178419 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.178352    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bbjrk\""
Sep 17 18:40:55.178617 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.178602    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 18:40:55.178788 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.178773    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 18:40:55.182718 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.182677    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 18:40:55.213093 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.213053    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:55.244619 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.244583    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-metrics-client-ca\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.244817 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.244630    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-root\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.244817 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.244731    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.244817 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.244774    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-sys\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.245012 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.244823    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-textfile\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.245012 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.244875    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scnm6\" (UniqueName: \"kubernetes.io/projected/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-kube-api-access-scnm6\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.245012 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.244922    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-tls\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.245012 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.244970    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-wtmp\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.245012 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.244995    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-accelerators-collector-config\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.309896 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.309667    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:55.353069 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.352387    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-metrics-client-ca\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.353069 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.352437    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-root\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.353069 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.352478    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.353069 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.352506    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-sys\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.353069 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.352562    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-textfile\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.353069 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.352590    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scnm6\" (UniqueName: \"kubernetes.io/projected/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-kube-api-access-scnm6\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.353069 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.352633    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-tls\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.353069 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.352661    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-wtmp\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.353069 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.352686    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-accelerators-collector-config\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.353674 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.353383    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-accelerators-collector-config\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.354484 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.354441    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-metrics-client-ca\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.354598 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.354517    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-root\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.355322 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.355212    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-sys\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.355322 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:55.355319    2587 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 18:40:55.355482 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:40:55.355390    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-tls podName:3a8d6c0f-51b4-4eda-a99f-0708f00bd53d nodeName:}" failed. No retries permitted until 2026-09-17 18:40:55.855370013 +0000 UTC m=+107.734338762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-tls") pod "node-exporter-wtwnk" (UID: "3a8d6c0f-51b4-4eda-a99f-0708f00bd53d") : secret "node-exporter-tls" not found
Sep 17 18:40:55.356769 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.356577    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-wtmp\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.356883 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.356815    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-textfile\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.364706 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.364495    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.372397 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.372114    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scnm6\" (UniqueName: \"kubernetes.io/projected/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-kube-api-access-scnm6\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.406453 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.406373    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:55.505143 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.505092    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:55.604798 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.604667    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:55.859287 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.858431    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-tls\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.861800 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.861771    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a8d6c0f-51b4-4eda-a99f-0708f00bd53d-node-exporter-tls\") pod \"node-exporter-wtwnk\" (UID: \"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d\") " pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:55.964790 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:55.964716    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:56.063663 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:56.063629    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:56.094982 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:56.094702    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wtwnk"
Sep 17 18:40:56.230288 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:56.227613    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:56.529474 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:56.529405    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:56.744525 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:40:56.744494    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8d6c0f_51b4_4eda_a99f_0708f00bd53d.slice/crio-1f0d340b5bee4eb1392ab9cd9198775d22086f940b3f5a34abb3203a40dace06 WatchSource:0}: Error finding container 1f0d340b5bee4eb1392ab9cd9198775d22086f940b3f5a34abb3203a40dace06: Status 404 returned error can't find the container with id 1f0d340b5bee4eb1392ab9cd9198775d22086f940b3f5a34abb3203a40dace06
Sep 17 18:40:56.810328 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:56.810233    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wtwnk" event={"ID":"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d","Type":"ContainerStarted","Data":"1f0d340b5bee4eb1392ab9cd9198775d22086f940b3f5a34abb3203a40dace06"}
Sep 17 18:40:57.463842 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:57.463101    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5894f4d576-l4h7z"]
Sep 17 18:40:57.587923 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:57.587889    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:57.709124 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:57.708995    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:57.804905 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:57.804849    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:57.820545 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:57.820509    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9r5ml" event={"ID":"c65d126d-844f-4c22-b0c8-3747a48aa285","Type":"ContainerStarted","Data":"94a0d7b3a694bf918e11722567ec21fca7445f4b79aba9114d580c8c1dd1f20b"}
Sep 17 18:40:57.820976 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:57.820563    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9r5ml" event={"ID":"c65d126d-844f-4c22-b0c8-3747a48aa285","Type":"ContainerStarted","Data":"35028d851c8a491ae3dae5a0599f10bf9a5e9caa0b908d08a42703d43d954c62"}
Sep 17 18:40:57.820976 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:57.820691    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:40:57.900414 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:57.899403    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:58.159006 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:58.158972    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:58.269653 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:58.269616    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:58.543874 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:58.543836    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:58.827567 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:58.827485    2587 generic.go:356] "Generic (PLEG): container finished" podID="3a8d6c0f-51b4-4eda-a99f-0708f00bd53d" containerID="fe7905c70dd5ffd5dbdefaa6f138ef505589cc8e033f24141022611b189c99bf" exitCode=0
Sep 17 18:40:58.827733 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:58.827556    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wtwnk" event={"ID":"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d","Type":"ContainerDied","Data":"fe7905c70dd5ffd5dbdefaa6f138ef505589cc8e033f24141022611b189c99bf"}
Sep 17 18:40:58.863115 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:58.863067    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9r5ml" podStartSLOduration=72.000872298 podStartE2EDuration="1m14.863043168s" totalImagesPullingTime="2.86217087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:57.843117465 +0000 UTC m=+109.722086223" watchObservedRunningTime="2026-09-17 18:40:58.863043168 +0000 UTC m=+110.742011925"
Sep 17 18:40:58.897040 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:58.897008    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:59.600436 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:59.600399    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:59.704891 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:59.704856    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:59.835563 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:59.835529    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wtwnk" event={"ID":"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d","Type":"ContainerStarted","Data":"46a3b8bf96e9752ce9cd17466a41d15d0f23ea0906ecaf86ff64b55ac6e79653"}
Sep 17 18:40:59.835811 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:59.835786    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wtwnk" event={"ID":"3a8d6c0f-51b4-4eda-a99f-0708f00bd53d","Type":"ContainerStarted","Data":"ccc106c9574797c37368ad6c332d251d7c17ebe701c6bb59bff313d6a779b4cc"}
Sep 17 18:40:59.866447 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:40:59.866363    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:00.212343 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:00.212259    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:00.434562 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:00.434524    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:00.517605 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:00.517572    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:01.583002 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:01.582964    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:05.705476 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:05.705425    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:05.821066 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:05.821038    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:07.830432 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:07.830404    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9r5ml"
Sep 17 18:41:07.851495 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:07.851436    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wtwnk" podStartSLOduration=10.98708721 podStartE2EDuration="12.851418908s" totalImagesPullingTime="1.769723334s" totalInitContainerRuntime="94.608364ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:59.903537527 +0000 UTC m=+111.782506296" watchObservedRunningTime="2026-09-17 18:41:07.851418908 +0000 UTC m=+119.730387860"
Sep 17 18:41:08.184702 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:08.184611    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:08.285346 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:08.285312    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:11.108229 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:11.108186    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:11.208016 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:11.207975    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:14.774665 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:14.774626    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:14.864022 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:14.863984    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:15.525467 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:15.525431    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:15.615662 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:15.615625    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:15.798343 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:15.798251    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:17.113802 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:17.113734    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:17.381095 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:17.381014    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:17.469013 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:17.468931    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:41:19.984171 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:19.984131    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs\") pod \"network-metrics-daemon-7m9j8\" (UID: \"b425f704-e370-447b-a2f4-23d81057111d\") " pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:41:19.986750 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:19.986726    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b425f704-e370-447b-a2f4-23d81057111d-metrics-certs\") pod \"network-metrics-daemon-7m9j8\" (UID: \"b425f704-e370-447b-a2f4-23d81057111d\") " pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:41:20.267323 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:20.267236    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7pg8w\""
Sep 17 18:41:20.275406 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:20.275377    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7m9j8"
Sep 17 18:41:20.571911 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:20.571888    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7m9j8"]
Sep 17 18:41:20.574045 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:41:20.574014    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb425f704_e370_447b_a2f4_23d81057111d.slice/crio-c33f3676062214ed315910d3196ddf9d56c1b5377fe9c48e81e2d0cba3332495 WatchSource:0}: Error finding container c33f3676062214ed315910d3196ddf9d56c1b5377fe9c48e81e2d0cba3332495: Status 404 returned error can't find the container with id c33f3676062214ed315910d3196ddf9d56c1b5377fe9c48e81e2d0cba3332495
Sep 17 18:41:20.924093 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:20.924008    2587 generic.go:356] "Generic (PLEG): container finished" podID="9e5a7245-d5c1-4419-9f7c-d3b5bdea88da" containerID="2f5db043d4d4db24985e1c7845ce1006302508bb8543e211d3cb8e4e5c6fb62d" exitCode=0
Sep 17 18:41:20.924093 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:20.924071    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k" event={"ID":"9e5a7245-d5c1-4419-9f7c-d3b5bdea88da","Type":"ContainerDied","Data":"2f5db043d4d4db24985e1c7845ce1006302508bb8543e211d3cb8e4e5c6fb62d"}
Sep 17 18:41:20.924426 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:20.924410    2587 scope.go:117] "RemoveContainer" containerID="2f5db043d4d4db24985e1c7845ce1006302508bb8543e211d3cb8e4e5c6fb62d"
Sep 17 18:41:20.925368 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:20.925346    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7m9j8" event={"ID":"b425f704-e370-447b-a2f4-23d81057111d","Type":"ContainerStarted","Data":"c33f3676062214ed315910d3196ddf9d56c1b5377fe9c48e81e2d0cba3332495"}
Sep 17 18:41:21.932313 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:21.932224    2587 generic.go:356] "Generic (PLEG): container finished" podID="2af1f706-2aa2-4b4a-9960-1eb443e3d40e" containerID="9b49f88b3012aaaa60e90ca44905ff9dea7cdcf7a65a6038b8d13574e8f86a49" exitCode=0
Sep 17 18:41:21.932742 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:21.932309    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd" event={"ID":"2af1f706-2aa2-4b4a-9960-1eb443e3d40e","Type":"ContainerDied","Data":"9b49f88b3012aaaa60e90ca44905ff9dea7cdcf7a65a6038b8d13574e8f86a49"}
Sep 17 18:41:21.932803 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:21.932777    2587 scope.go:117] "RemoveContainer" containerID="9b49f88b3012aaaa60e90ca44905ff9dea7cdcf7a65a6038b8d13574e8f86a49"
Sep 17 18:41:21.934428 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:21.934397    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7m9j8" event={"ID":"b425f704-e370-447b-a2f4-23d81057111d","Type":"ContainerStarted","Data":"9ec8cf2f0ed416b451180c6f4abe64d9f3caa89882796d446b2f1d6ebdb37efb"}
Sep 17 18:41:21.934526 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:21.934468    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7m9j8" event={"ID":"b425f704-e370-447b-a2f4-23d81057111d","Type":"ContainerStarted","Data":"284f3a2592d47d3267e83f55b769a8aa3e1823567c1f00eeed4ff4a42d4bbd13"}
Sep 17 18:41:21.936576 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:21.936557    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-f6j6k" event={"ID":"9e5a7245-d5c1-4419-9f7c-d3b5bdea88da","Type":"ContainerStarted","Data":"8e93f7fcba1ef619bafdf3cebbd91ee56044db1fc20092e5671245a1223897cf"}
Sep 17 18:41:22.061031 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.060979    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7m9j8" podStartSLOduration=130.972602052 podStartE2EDuration="2m12.06096276s" totalImagesPullingTime="1.088360708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:22.05932246 +0000 UTC m=+133.938291230" watchObservedRunningTime="2026-09-17 18:41:22.06096276 +0000 UTC m=+133.939931514"
Sep 17 18:41:22.201011 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.200917    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:22.343439 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.343407    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:22.496549 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.496502    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5894f4d576-l4h7z" podUID="5b9d9afe-fb1a-4eec-9e77-9e9675ea2421" containerName="registry" containerID="cri-o://993328358add779d983dfd0c12a4a56456d9733231b824b23b5ed5719cfb883f" gracePeriod=30
Sep 17 18:41:22.743179 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.743150    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:41:22.805573 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.805494    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-bound-sa-token\") pod \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") "
Sep 17 18:41:22.805573 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.805561    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-installation-pull-secrets\") pod \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") "
Sep 17 18:41:22.805773 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.805578    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-image-registry-private-configuration\") pod \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") "
Sep 17 18:41:22.805773 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.805753    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-ca-trust-extracted\") pod \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") "
Sep 17 18:41:22.805887 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.805807    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqnfj\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-kube-api-access-cqnfj\") pod \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") "
Sep 17 18:41:22.805887 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.805844    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-certificates\") pod \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") "
Sep 17 18:41:22.806009 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.805910    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls\") pod \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") "
Sep 17 18:41:22.806009 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.805974    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-trusted-ca\") pod \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\" (UID: \"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421\") "
Sep 17 18:41:22.806744 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.806688    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:22.806744 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.806695    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:22.808767 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.808712    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:22.809034 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.808975    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:22.809318 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.809292    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:22.809424 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.809306    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-kube-api-access-cqnfj" (OuterVolumeSpecName: "kube-api-access-cqnfj") pod "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421"). InnerVolumeSpecName "kube-api-access-cqnfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:22.809424 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.809333    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:22.814822 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.814798    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421" (UID: "5b9d9afe-fb1a-4eec-9e77-9e9675ea2421"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:22.906877 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.906829    2587 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-tls\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:41:22.906877 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.906867    2587 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-trusted-ca\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:41:22.906877 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.906875    2587 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-bound-sa-token\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:41:22.906877 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.906883    2587 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-installation-pull-secrets\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:41:22.906877 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.906894    2587 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-image-registry-private-configuration\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:41:22.907211 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.906903    2587 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-ca-trust-extracted\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:41:22.907211 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.906911    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqnfj\" (UniqueName: \"kubernetes.io/projected/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-kube-api-access-cqnfj\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:41:22.907211 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.906919    2587 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421-registry-certificates\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:41:22.942842 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.942808    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-xdqjd" event={"ID":"2af1f706-2aa2-4b4a-9960-1eb443e3d40e","Type":"ContainerStarted","Data":"db5cfb894d4e938c99b0d6ca4afc71ee4419a768ae0aefe4290e5dcd2de99c9c"}
Sep 17 18:41:22.944225 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.944201    2587 generic.go:356] "Generic (PLEG): container finished" podID="5b9d9afe-fb1a-4eec-9e77-9e9675ea2421" containerID="993328358add779d983dfd0c12a4a56456d9733231b824b23b5ed5719cfb883f" exitCode=0
Sep 17 18:41:22.944355 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.944243    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5894f4d576-l4h7z"
Sep 17 18:41:22.944355 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.944276    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5894f4d576-l4h7z" event={"ID":"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421","Type":"ContainerDied","Data":"993328358add779d983dfd0c12a4a56456d9733231b824b23b5ed5719cfb883f"}
Sep 17 18:41:22.944355 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.944322    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5894f4d576-l4h7z" event={"ID":"5b9d9afe-fb1a-4eec-9e77-9e9675ea2421","Type":"ContainerDied","Data":"8ef48ce238ffdea8330c04f1495f76dc4d0001998a4c3287c3ab6cd1cd238259"}
Sep 17 18:41:22.944355 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.944343    2587 scope.go:117] "RemoveContainer" containerID="993328358add779d983dfd0c12a4a56456d9733231b824b23b5ed5719cfb883f"
Sep 17 18:41:22.953204 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.953187    2587 scope.go:117] "RemoveContainer" containerID="993328358add779d983dfd0c12a4a56456d9733231b824b23b5ed5719cfb883f"
Sep 17 18:41:22.953474 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:41:22.953458    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993328358add779d983dfd0c12a4a56456d9733231b824b23b5ed5719cfb883f\": container with ID starting with 993328358add779d983dfd0c12a4a56456d9733231b824b23b5ed5719cfb883f not found: ID does not exist" containerID="993328358add779d983dfd0c12a4a56456d9733231b824b23b5ed5719cfb883f"
Sep 17 18:41:22.953524 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.953481    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993328358add779d983dfd0c12a4a56456d9733231b824b23b5ed5719cfb883f"} err="failed to get container status \"993328358add779d983dfd0c12a4a56456d9733231b824b23b5ed5719cfb883f\": rpc error: code = NotFound desc = could not find container \"993328358add779d983dfd0c12a4a56456d9733231b824b23b5ed5719cfb883f\": container with ID starting with 993328358add779d983dfd0c12a4a56456d9733231b824b23b5ed5719cfb883f not found: ID does not exist"
Sep 17 18:41:22.997983 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:22.997933    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5894f4d576-l4h7z"]
Sep 17 18:41:23.011258 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:23.011226    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5894f4d576-l4h7z"]
Sep 17 18:41:23.241053 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:23.241017    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:23.384515 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:23.384472    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:24.258033 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:24.258002    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b9d9afe-fb1a-4eec-9e77-9e9675ea2421" path="/var/lib/kubelet/pods/5b9d9afe-fb1a-4eec-9e77-9e9675ea2421/volumes"
Sep 17 18:41:38.164127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:38.164088    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:38.264337 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:38.264303    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:42.024753 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:42.024720    2587 generic.go:356] "Generic (PLEG): container finished" podID="3ba38230-b61f-4204-8b35-d60d7ee7b3b2" containerID="ec6dc94b7ec251f3e9f2092f040086328c1e808267609020de12850f4b9a0c07" exitCode=0
Sep 17 18:41:42.025262 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:42.024797    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp" event={"ID":"3ba38230-b61f-4204-8b35-d60d7ee7b3b2","Type":"ContainerDied","Data":"ec6dc94b7ec251f3e9f2092f040086328c1e808267609020de12850f4b9a0c07"}
Sep 17 18:41:42.025262 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:42.025207    2587 scope.go:117] "RemoveContainer" containerID="ec6dc94b7ec251f3e9f2092f040086328c1e808267609020de12850f4b9a0c07"
Sep 17 18:41:43.030456 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:41:43.030417    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-dn9jp" event={"ID":"3ba38230-b61f-4204-8b35-d60d7ee7b3b2","Type":"ContainerStarted","Data":"5d9d904b252ec2d1a074a165e81320814f11d83377ab2aaa47f0cf3d11e1609d"}
Sep 17 18:42:14.585708 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:14.585661    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:15.271610 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:15.271572    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:18.876475 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:18.876439    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:24.306983 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:24.306899    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:24.370215 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:24.370181    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:24.448295 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:24.448250    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:24.511832 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:24.511798    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:25.043667 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:25.043624    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:25.138763 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:25.138714    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:29.922777 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:29.922724    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:30.023792 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:30.023754    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:31.661529 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:31.661487    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:35.381234 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:35.381194    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:35.481669 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:35.481637    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:35.699531 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:35.699432    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:35.762036 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:35.761989    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:36.307373 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:36.307319    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:36.374553 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:36.374516    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:37.296840 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:37.296799    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:37.377875 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:37.377840    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:41.863659 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:41.863625    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:41.931585 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:41.931542    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:46.390743 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:46.390700    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:46.491433 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:46.491403    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:46.594228 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:46.594188    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:49.423889 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:49.423850    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:52.891631 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:52.891594    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:52.970970 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:52.970906    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:53.172889 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:53.172797    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:53.271441 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:42:53.271407    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:02.812638 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:02.812605    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:10.133600 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:10.133576    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 18:44:10.135627 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:10.135606    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 18:44:10.144115 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:10.144084    2587 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:44:20.359173 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:20.359114    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:20.458308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:20.458271    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:20.516681 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:20.516649    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:37.575895 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:37.575858    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:42.978071 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:42.978030    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:43.077231 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:43.077192    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:48.605995 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:48.605934    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:48.881284 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:48.881197    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:49.058822 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:49.058786    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:49.143310 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:49.143242    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:49.493113 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:49.493072    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:49.689208 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:49.689178    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:49.839125 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:49.839021    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:49.938218 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:49.938191    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:50.148912 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:50.148850    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:50.238922 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:50.238901    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:50.339873 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:50.339854    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:50.439829 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:50.439778    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:50.539485 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:44:50.539457    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:07.997337 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:07.997286    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:08.077260 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:08.077230    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:41.698880 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:41.698844    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:41.799575 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:41.799538    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:43.122707 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:43.122672    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:43.234024 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:43.233992    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:43.299639 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:43.299612    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:43.793986 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:43.793929    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:43.895372 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:43.895343    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:52.902421 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:52.902382    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:53.781315 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:53.781279    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:54.222705 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:54.222608    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:54.359125 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:54.359079    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:54.458780 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:54.458748    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:56.695139 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:56.695081    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:56.794711 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:56.794678    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:58.203805 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:58.203769    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:58.303333 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:45:58.303292    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:00.264192 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:00.264154    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:00.724845 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:00.724749    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:01.687406 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:01.687368    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:01.789891 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:01.789851    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:03.186660 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:03.186625    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:03.293464 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:03.293427    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:03.391604 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:03.391560    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:09.401610 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:09.401572    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:12.700508 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:12.700468    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:12.799700 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:12.799662    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:14.383867 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:14.383832    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:14.459053 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:14.459005    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:21.788803 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:21.788704    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:27.026367 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:27.026333    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:27.109511 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:27.109475    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:28.403588 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:28.403555    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:30.016304 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:30.016263    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:31.520680 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:31.520638    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:31.620544 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:31.620511    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:35.769678 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:35.769629    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:35.833025 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:35.832982    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:39.353587 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:39.353547    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:41.253583 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:41.253543    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:41.365605 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:41.365574    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:46.399768 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.399736    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:46.555374 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.555333    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:46.566474 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.566438    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"]
Sep 17 18:46:46.566739 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.566727    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b9d9afe-fb1a-4eec-9e77-9e9675ea2421" containerName="registry"
Sep 17 18:46:46.566791 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.566742    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b9d9afe-fb1a-4eec-9e77-9e9675ea2421" containerName="registry"
Sep 17 18:46:46.566826 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.566805    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b9d9afe-fb1a-4eec-9e77-9e9675ea2421" containerName="registry"
Sep 17 18:46:46.568919 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.568904    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:46.572354 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.572320    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 18:46:46.572613 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.572594    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fn5xz\""
Sep 17 18:46:46.573233 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.573218    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 18:46:46.573587 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.573561    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:46:46.573693 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.573623    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 18:46:46.573693 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.573633    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 18:46:46.581975 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.581918    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"]
Sep 17 18:46:46.684675 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.684585    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/130a5513-c37f-4691-b6b8-74d817accee3-metrics-cert\") pod \"lws-controller-manager-54799454dd-khwgm\" (UID: \"130a5513-c37f-4691-b6b8-74d817accee3\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:46.684675 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.684623    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/130a5513-c37f-4691-b6b8-74d817accee3-manager-config\") pod \"lws-controller-manager-54799454dd-khwgm\" (UID: \"130a5513-c37f-4691-b6b8-74d817accee3\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:46.684675 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.684650    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/130a5513-c37f-4691-b6b8-74d817accee3-cert\") pod \"lws-controller-manager-54799454dd-khwgm\" (UID: \"130a5513-c37f-4691-b6b8-74d817accee3\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:46.684927 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.684748    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwxk6\" (UniqueName: \"kubernetes.io/projected/130a5513-c37f-4691-b6b8-74d817accee3-kube-api-access-jwxk6\") pod \"lws-controller-manager-54799454dd-khwgm\" (UID: \"130a5513-c37f-4691-b6b8-74d817accee3\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:46.785127 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.785094    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/130a5513-c37f-4691-b6b8-74d817accee3-cert\") pod \"lws-controller-manager-54799454dd-khwgm\" (UID: \"130a5513-c37f-4691-b6b8-74d817accee3\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:46.785325 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.785136    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwxk6\" (UniqueName: \"kubernetes.io/projected/130a5513-c37f-4691-b6b8-74d817accee3-kube-api-access-jwxk6\") pod \"lws-controller-manager-54799454dd-khwgm\" (UID: \"130a5513-c37f-4691-b6b8-74d817accee3\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:46.785325 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.785194    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/130a5513-c37f-4691-b6b8-74d817accee3-metrics-cert\") pod \"lws-controller-manager-54799454dd-khwgm\" (UID: \"130a5513-c37f-4691-b6b8-74d817accee3\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:46.785325 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.785211    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/130a5513-c37f-4691-b6b8-74d817accee3-manager-config\") pod \"lws-controller-manager-54799454dd-khwgm\" (UID: \"130a5513-c37f-4691-b6b8-74d817accee3\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:46.785805 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.785781    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/130a5513-c37f-4691-b6b8-74d817accee3-manager-config\") pod \"lws-controller-manager-54799454dd-khwgm\" (UID: \"130a5513-c37f-4691-b6b8-74d817accee3\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:46.787783 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.787760    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/130a5513-c37f-4691-b6b8-74d817accee3-metrics-cert\") pod \"lws-controller-manager-54799454dd-khwgm\" (UID: \"130a5513-c37f-4691-b6b8-74d817accee3\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:46.787878 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.787833    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/130a5513-c37f-4691-b6b8-74d817accee3-cert\") pod \"lws-controller-manager-54799454dd-khwgm\" (UID: \"130a5513-c37f-4691-b6b8-74d817accee3\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:46.794696 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.794675    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwxk6\" (UniqueName: \"kubernetes.io/projected/130a5513-c37f-4691-b6b8-74d817accee3-kube-api-access-jwxk6\") pod \"lws-controller-manager-54799454dd-khwgm\" (UID: \"130a5513-c37f-4691-b6b8-74d817accee3\") " pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:46.878390 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:46.878346    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:47.163613 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:47.163580    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"]
Sep 17 18:46:47.167921 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:46:47.167897    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130a5513_c37f_4691_b6b8_74d817accee3.slice/crio-70e4b7d785a18b658c383bd0642fa865d625af8a263126d24b1abf468f1e0251 WatchSource:0}: Error finding container 70e4b7d785a18b658c383bd0642fa865d625af8a263126d24b1abf468f1e0251: Status 404 returned error can't find the container with id 70e4b7d785a18b658c383bd0642fa865d625af8a263126d24b1abf468f1e0251
Sep 17 18:46:47.169825 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:47.169809    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:46:47.216558 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:47.216520    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm" event={"ID":"130a5513-c37f-4691-b6b8-74d817accee3","Type":"ContainerStarted","Data":"70e4b7d785a18b658c383bd0642fa865d625af8a263126d24b1abf468f1e0251"}
Sep 17 18:46:50.233763 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:50.233731    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm" event={"ID":"130a5513-c37f-4691-b6b8-74d817accee3","Type":"ContainerStarted","Data":"5e56ff17744b1805e5f25e95eb3c195b6322c1804e6c5bbbcefe2f1b195874a8"}
Sep 17 18:46:50.234206 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:50.233792    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:46:50.258707 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:50.258656    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm" podStartSLOduration=1.355365761 podStartE2EDuration="4.25864085s" totalImagesPullingTime="2.903275089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:50.256370681 +0000 UTC m=+462.135339434" watchObservedRunningTime="2026-09-17 18:46:50.25864085 +0000 UTC m=+462.137609606"
Sep 17 18:46:51.469422 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:51.469388    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:51.569447 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:51.569412    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:51.664218 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:46:51.664185    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:01.240510 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:01.240474    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-54799454dd-khwgm"
Sep 17 18:47:07.625907 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:07.625857    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:10.027694 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:10.027652    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:11.711474 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:11.711430    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:12.087415 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:12.087359    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:12.332069 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:12.332035    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:13.165630 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:13.165594    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:14.508668 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:14.508629    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:15.245475 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:15.245438    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:15.341792 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:15.341760    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:15.613553 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:15.613475    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:15.721483 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:15.721443    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:16.013322 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:16.013290    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:16.112672 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:16.112643    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:21.118087 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:21.118046    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:21.312475 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:21.312437    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:24.256566 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:24.256527    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:24.358353 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:24.358315    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:26.570496 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:26.570463    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:26.707131 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:26.707095    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:27.770101 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:27.770070    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:27.875640 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:27.875595    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:29.341604 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:29.341559    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:29.508045 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:29.508014    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:29.766663 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:29.766634    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:33.983576 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:33.983539    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:34.135529 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:34.135484    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:44.296000 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:44.295921    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:44.395862 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:44.395821    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:47.390516 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:47.390479    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:47.487884 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:47.487822    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:52.496730 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:52.496684    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:52.597343 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:52.597293    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:57.285725 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:57.285688    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:57.384321 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:57.384290    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:57.485431 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:57.485385    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:59.409461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:59.409413    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:59.510632 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:47:59.510606    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:09.423258 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.423225    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:09.535765 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.535731    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:09.598948 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.598918    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:09.796074 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.796043    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5tjqp"]
Sep 17 18:48:09.799215 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.799198    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-5tjqp"
Sep 17 18:48:09.802184 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.802160    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 18:48:09.802297 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.802275    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 18:48:09.803198 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.803180    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wm6w4\""
Sep 17 18:48:09.803295 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.803270    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 18:48:09.812897 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.812868    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5tjqp"]
Sep 17 18:48:09.846454 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.846423    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5tjqp"]
Sep 17 18:48:09.853442 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.853421    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/635a6392-7e09-4b14-8210-4cd7f8a88197-config-file\") pod \"limitador-limitador-84994f7ffc-5tjqp\" (UID: \"635a6392-7e09-4b14-8210-4cd7f8a88197\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5tjqp"
Sep 17 18:48:09.853509 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.853463    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnxbc\" (UniqueName: \"kubernetes.io/projected/635a6392-7e09-4b14-8210-4cd7f8a88197-kube-api-access-jnxbc\") pod \"limitador-limitador-84994f7ffc-5tjqp\" (UID: \"635a6392-7e09-4b14-8210-4cd7f8a88197\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5tjqp"
Sep 17 18:48:09.954322 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.954290    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnxbc\" (UniqueName: \"kubernetes.io/projected/635a6392-7e09-4b14-8210-4cd7f8a88197-kube-api-access-jnxbc\") pod \"limitador-limitador-84994f7ffc-5tjqp\" (UID: \"635a6392-7e09-4b14-8210-4cd7f8a88197\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5tjqp"
Sep 17 18:48:09.954492 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.954358    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/635a6392-7e09-4b14-8210-4cd7f8a88197-config-file\") pod \"limitador-limitador-84994f7ffc-5tjqp\" (UID: \"635a6392-7e09-4b14-8210-4cd7f8a88197\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5tjqp"
Sep 17 18:48:09.955077 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.955052    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/635a6392-7e09-4b14-8210-4cd7f8a88197-config-file\") pod \"limitador-limitador-84994f7ffc-5tjqp\" (UID: \"635a6392-7e09-4b14-8210-4cd7f8a88197\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5tjqp"
Sep 17 18:48:09.965083 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:09.965050    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnxbc\" (UniqueName: \"kubernetes.io/projected/635a6392-7e09-4b14-8210-4cd7f8a88197-kube-api-access-jnxbc\") pod \"limitador-limitador-84994f7ffc-5tjqp\" (UID: \"635a6392-7e09-4b14-8210-4cd7f8a88197\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5tjqp"
Sep 17 18:48:10.111354 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:10.111276    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wm6w4\""
Sep 17 18:48:10.119467 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:10.119443    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-5tjqp"
Sep 17 18:48:10.273486 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:10.273458    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:10.380854 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:10.380779    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:10.397287 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:10.397260    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5tjqp"]
Sep 17 18:48:10.401557 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:48:10.401529    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635a6392_7e09_4b14_8210_4cd7f8a88197.slice/crio-40cb8355009661a55bee3bb630c80e522151c228a48c334e69391fc76956220b WatchSource:0}: Error finding container 40cb8355009661a55bee3bb630c80e522151c228a48c334e69391fc76956220b: Status 404 returned error can't find the container with id 40cb8355009661a55bee3bb630c80e522151c228a48c334e69391fc76956220b
Sep 17 18:48:10.547541 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:10.547509    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-5tjqp" event={"ID":"635a6392-7e09-4b14-8210-4cd7f8a88197","Type":"ContainerStarted","Data":"40cb8355009661a55bee3bb630c80e522151c228a48c334e69391fc76956220b"}
Sep 17 18:48:14.564757 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:14.564716    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-5tjqp" event={"ID":"635a6392-7e09-4b14-8210-4cd7f8a88197","Type":"ContainerStarted","Data":"dad64a41c976596bc92deae95f448230630aaf231687c0a0cdc0cb8a706394e2"}
Sep 17 18:48:14.565229 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:14.564869    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-5tjqp"
Sep 17 18:48:14.582611 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:14.582556    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-5tjqp" podStartSLOduration=2.408309485 podStartE2EDuration="5.58253837s" totalImagesPullingTime="3.174228885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:14.58134891 +0000 UTC m=+546.460317668" watchObservedRunningTime="2026-09-17 18:48:14.58253837 +0000 UTC m=+546.461507145"
Sep 17 18:48:25.570241 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:25.570211    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-5tjqp"
Sep 17 18:48:31.703738 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:31.703695    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:32.262041 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:32.262010    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:40.838066 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:40.838029    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:40.950176 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:40.950142    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:58.373458 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:58.373414    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:58.521295 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:48:58.521259    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:10.162390 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:49:10.162359    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 18:49:10.162819 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:49:10.162573    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 18:49:12.762991 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:49:12.762926    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:12.872718 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:49:12.872684    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:12.968106 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:49:12.968074    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:13.068417 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:49:13.068347    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:13.170536 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:49:13.170513    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:13.596823 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:49:13.596791    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:13.705158 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:49:13.705133    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:13.805348 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:49:13.805318    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:13.904984 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:49:13.904912    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:40.774553 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:49:40.774513    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:42.122081 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:49:42.122043    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:30.733309 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:30.733266    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:30.847394 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:30.847354    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:30.938265 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:30.938224    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:31.323826 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:31.323780    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:32.122364 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:32.122324    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:32.234641 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:32.234602    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:40.843079 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:40.843035    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:47.428031 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.427994    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-cmwf4"]
Sep 17 18:50:47.431234 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.431206    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cmwf4"
Sep 17 18:50:47.433696 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.433674    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:50:47.434739 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.434718    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4wtpq\""
Sep 17 18:50:47.434837 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.434722    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:50:47.434837 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.434724    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:50:47.434837 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.434727    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 18:50:47.439674 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.439651    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cmwf4"]
Sep 17 18:50:47.555805 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.555763    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc6ce9e-444f-4d6a-a586-5bab6eba20b4-model-cache\") pod \"s3-init-cmwf4\" (UID: \"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4\") " pod="kserve/s3-init-cmwf4"
Sep 17 18:50:47.556006 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.555828    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f5ll\" (UniqueName: \"kubernetes.io/projected/5fc6ce9e-444f-4d6a-a586-5bab6eba20b4-kube-api-access-4f5ll\") pod \"s3-init-cmwf4\" (UID: \"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4\") " pod="kserve/s3-init-cmwf4"
Sep 17 18:50:47.657033 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.656979    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f5ll\" (UniqueName: \"kubernetes.io/projected/5fc6ce9e-444f-4d6a-a586-5bab6eba20b4-kube-api-access-4f5ll\") pod \"s3-init-cmwf4\" (UID: \"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4\") " pod="kserve/s3-init-cmwf4"
Sep 17 18:50:47.657196 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.657066    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc6ce9e-444f-4d6a-a586-5bab6eba20b4-model-cache\") pod \"s3-init-cmwf4\" (UID: \"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4\") " pod="kserve/s3-init-cmwf4"
Sep 17 18:50:47.657387 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.657367    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc6ce9e-444f-4d6a-a586-5bab6eba20b4-model-cache\") pod \"s3-init-cmwf4\" (UID: \"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4\") " pod="kserve/s3-init-cmwf4"
Sep 17 18:50:47.665299 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.665278    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f5ll\" (UniqueName: \"kubernetes.io/projected/5fc6ce9e-444f-4d6a-a586-5bab6eba20b4-kube-api-access-4f5ll\") pod \"s3-init-cmwf4\" (UID: \"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4\") " pod="kserve/s3-init-cmwf4"
Sep 17 18:50:47.741301 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:47.741276    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cmwf4"
Sep 17 18:50:48.011008 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:48.010917    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cmwf4"]
Sep 17 18:50:48.013249 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:50:48.013210    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fc6ce9e_444f_4d6a_a586_5bab6eba20b4.slice/crio-8b7bb66714db8bd0bd76e2a1f4689544ab6028492d02000d28d42f5f216a7167 WatchSource:0}: Error finding container 8b7bb66714db8bd0bd76e2a1f4689544ab6028492d02000d28d42f5f216a7167: Status 404 returned error can't find the container with id 8b7bb66714db8bd0bd76e2a1f4689544ab6028492d02000d28d42f5f216a7167
Sep 17 18:50:48.182379 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:48.182341    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cmwf4" event={"ID":"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4","Type":"ContainerStarted","Data":"8b7bb66714db8bd0bd76e2a1f4689544ab6028492d02000d28d42f5f216a7167"}
Sep 17 18:50:52.203233 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:52.203198    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cmwf4" event={"ID":"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4","Type":"ContainerStarted","Data":"96eafe12462dca9b8996a92c6560bec382aa318be04613de12bcc0f5cb0b3d47"}
Sep 17 18:50:52.484171 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:52.484146    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:53.209257 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:53.209219    2587 generic.go:356] "Generic (PLEG): container finished" podID="5fc6ce9e-444f-4d6a-a586-5bab6eba20b4" containerID="96eafe12462dca9b8996a92c6560bec382aa318be04613de12bcc0f5cb0b3d47" exitCode=0
Sep 17 18:50:53.209662 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:53.209300    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cmwf4" event={"ID":"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4","Type":"ContainerDied","Data":"96eafe12462dca9b8996a92c6560bec382aa318be04613de12bcc0f5cb0b3d47"}
Sep 17 18:50:58.236418 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:58.236384    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cmwf4" event={"ID":"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4","Type":"ContainerStarted","Data":"1e3b03f7615c155cf1c609e6e86ba46b81954707ef595a958174db1cc1042575"}
Sep 17 18:50:58.255805 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:50:58.255742    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-cmwf4" podStartSLOduration=1.981179707 podStartE2EDuration="11.255722122s" totalImagesPullingTime="7.541630937s" totalInitContainerRuntime="1.732911478s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:58.25324615 +0000 UTC m=+710.132214912" watchObservedRunningTime="2026-09-17 18:50:58.255722122 +0000 UTC m=+710.134690886"
Sep 17 18:51:07.888001 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:51:07.887966    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:59.643219 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:51:59.643185    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:03.504264 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:03.504231    2587 generic.go:356] "Generic (PLEG): container finished" podID="5fc6ce9e-444f-4d6a-a586-5bab6eba20b4" containerID="1e3b03f7615c155cf1c609e6e86ba46b81954707ef595a958174db1cc1042575" exitCode=0
Sep 17 18:52:03.504672 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:03.504298    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cmwf4" event={"ID":"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4","Type":"ContainerDied","Data":"1e3b03f7615c155cf1c609e6e86ba46b81954707ef595a958174db1cc1042575"}
Sep 17 18:52:04.642877 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:04.642844    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cmwf4"
Sep 17 18:52:04.804704 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:04.804625    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc6ce9e-444f-4d6a-a586-5bab6eba20b4-model-cache\") pod \"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4\" (UID: \"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4\") "
Sep 17 18:52:04.804704 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:04.804678    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f5ll\" (UniqueName: \"kubernetes.io/projected/5fc6ce9e-444f-4d6a-a586-5bab6eba20b4-kube-api-access-4f5ll\") pod \"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4\" (UID: \"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4\") "
Sep 17 18:52:04.807068 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:04.807044    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc6ce9e-444f-4d6a-a586-5bab6eba20b4-kube-api-access-4f5ll" (OuterVolumeSpecName: "kube-api-access-4f5ll") pod "5fc6ce9e-444f-4d6a-a586-5bab6eba20b4" (UID: "5fc6ce9e-444f-4d6a-a586-5bab6eba20b4"). InnerVolumeSpecName "kube-api-access-4f5ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:52:04.823748 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:04.823710    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc6ce9e-444f-4d6a-a586-5bab6eba20b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "5fc6ce9e-444f-4d6a-a586-5bab6eba20b4" (UID: "5fc6ce9e-444f-4d6a-a586-5bab6eba20b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:04.905623 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:04.905587    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc6ce9e-444f-4d6a-a586-5bab6eba20b4-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:52:04.905623 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:04.905617    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f5ll\" (UniqueName: \"kubernetes.io/projected/5fc6ce9e-444f-4d6a-a586-5bab6eba20b4-kube-api-access-4f5ll\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:52:05.519008 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:05.518963    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cmwf4" event={"ID":"5fc6ce9e-444f-4d6a-a586-5bab6eba20b4","Type":"ContainerDied","Data":"8b7bb66714db8bd0bd76e2a1f4689544ab6028492d02000d28d42f5f216a7167"}
Sep 17 18:52:05.519008 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:05.518995    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cmwf4"
Sep 17 18:52:05.519272 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:05.519020    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b7bb66714db8bd0bd76e2a1f4689544ab6028492d02000d28d42f5f216a7167"
Sep 17 18:52:05.881393 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:05.881306    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:05.984017 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:05.983979    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:18.080295 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:18.080256    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:18.180387 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:18.180353    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:27.935565 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:27.935465    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:28.397582 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.397520    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"]
Sep 17 18:52:28.397925 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.397907    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fc6ce9e-444f-4d6a-a586-5bab6eba20b4" containerName="s3-init"
Sep 17 18:52:28.398038 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.397928    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fc6ce9e-444f-4d6a-a586-5bab6eba20b4" containerName="s3-init"
Sep 17 18:52:28.398038 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.397971    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fc6ce9e-444f-4d6a-a586-5bab6eba20b4" containerName="download-hf-model"
Sep 17 18:52:28.398038 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.397982    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fc6ce9e-444f-4d6a-a586-5bab6eba20b4" containerName="download-hf-model"
Sep 17 18:52:28.398178 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.398091    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fc6ce9e-444f-4d6a-a586-5bab6eba20b4" containerName="s3-init"
Sep 17 18:52:28.400795 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.400776    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"
Sep 17 18:52:28.405169 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.405149    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 18:52:28.405285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.405171    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-t6wn5\""
Sep 17 18:52:28.405285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.405177    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 18:52:28.405285 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.405152    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-5rcdt\""
Sep 17 18:52:28.405419 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.405291    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 18:52:28.415095 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.415072    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"]
Sep 17 18:52:28.426785 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.426759    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:28.491666 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.491626    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47cae6de-3591-4321-87d6-51fa63e020bb-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb\" (UID: \"47cae6de-3591-4321-87d6-51fa63e020bb\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"
Sep 17 18:52:28.491849 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.491673    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n7kw\" (UniqueName: \"kubernetes.io/projected/47cae6de-3591-4321-87d6-51fa63e020bb-kube-api-access-8n7kw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb\" (UID: \"47cae6de-3591-4321-87d6-51fa63e020bb\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"
Sep 17 18:52:28.592760 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.592721    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47cae6de-3591-4321-87d6-51fa63e020bb-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb\" (UID: \"47cae6de-3591-4321-87d6-51fa63e020bb\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"
Sep 17 18:52:28.592760 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.592765    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n7kw\" (UniqueName: \"kubernetes.io/projected/47cae6de-3591-4321-87d6-51fa63e020bb-kube-api-access-8n7kw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb\" (UID: \"47cae6de-3591-4321-87d6-51fa63e020bb\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"
Sep 17 18:52:28.595479 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.595451    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47cae6de-3591-4321-87d6-51fa63e020bb-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb\" (UID: \"47cae6de-3591-4321-87d6-51fa63e020bb\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"
Sep 17 18:52:28.603196 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.603170    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n7kw\" (UniqueName: \"kubernetes.io/projected/47cae6de-3591-4321-87d6-51fa63e020bb-kube-api-access-8n7kw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb\" (UID: \"47cae6de-3591-4321-87d6-51fa63e020bb\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"
Sep 17 18:52:28.711653 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:28.711567    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"
Sep 17 18:52:29.008766 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:29.008742    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"]
Sep 17 18:52:29.010676 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:52:29.010649    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cae6de_3591_4321_87d6_51fa63e020bb.slice/crio-4d90c855190a0175d224fae6876611ba370ce42bc32c7330e388a5d7e62adda6 WatchSource:0}: Error finding container 4d90c855190a0175d224fae6876611ba370ce42bc32c7330e388a5d7e62adda6: Status 404 returned error can't find the container with id 4d90c855190a0175d224fae6876611ba370ce42bc32c7330e388a5d7e62adda6
Sep 17 18:52:29.013143 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:29.013126    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:52:29.615952 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:29.615901    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb" event={"ID":"47cae6de-3591-4321-87d6-51fa63e020bb","Type":"ContainerStarted","Data":"4d90c855190a0175d224fae6876611ba370ce42bc32c7330e388a5d7e62adda6"}
Sep 17 18:52:31.627088 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:31.626994    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb" event={"ID":"47cae6de-3591-4321-87d6-51fa63e020bb","Type":"ContainerStarted","Data":"9cab6ddc7cc9a69c44a042afa8e3c200790883c2be342f7a3b9dc19e9135966a"}
Sep 17 18:52:31.627480 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:31.627104    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"
Sep 17 18:52:31.646787 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:31.646735    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb" podStartSLOduration=1.380362142 podStartE2EDuration="3.646718262s" totalImagesPullingTime="2.26635612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:31.644473398 +0000 UTC m=+803.523442155" watchObservedRunningTime="2026-09-17 18:52:31.646718262 +0000 UTC m=+803.525687019"
Sep 17 18:52:36.934906 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:36.934860    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:37.244336 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:37.244298    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:51.516037 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:52:51.515998    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:02.000260 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:02.000225    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:02.101726 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:02.101693    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:02.633019 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:02.632987    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"
Sep 17 18:53:06.342549 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:06.342511    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:10.807500 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:10.807464    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"]
Sep 17 18:53:10.812883 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:10.812850    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:10.879612 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:10.879569    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:12.633238 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:12.633197    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb" podUID="47cae6de-3591-4321-87d6-51fa63e020bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:18.044433 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:18.044386    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:18.139193 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:18.139161    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:21.590538 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.590499    2587 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 17 18:53:21.590901 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:53:21.590537    2587 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb" podUID="47cae6de-3591-4321-87d6-51fa63e020bb" containerName="main" containerID="cri-o://9cab6ddc7cc9a69c44a042afa8e3c200790883c2be342f7a3b9dc19e9135966a"
Sep 17 18:53:21.590901 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.590602    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb" podUID="47cae6de-3591-4321-87d6-51fa63e020bb" containerName="main" containerID="cri-o://9cab6ddc7cc9a69c44a042afa8e3c200790883c2be342f7a3b9dc19e9135966a" gracePeriod=50
Sep 17 18:53:21.739361 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.739340    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb_47cae6de-3591-4321-87d6-51fa63e020bb/main/0.log"
Sep 17 18:53:21.741649 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.741633    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"
Sep 17 18:53:21.837190 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.837105    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n7kw\" (UniqueName: \"kubernetes.io/projected/47cae6de-3591-4321-87d6-51fa63e020bb-kube-api-access-8n7kw\") pod \"47cae6de-3591-4321-87d6-51fa63e020bb\" (UID: \"47cae6de-3591-4321-87d6-51fa63e020bb\") "
Sep 17 18:53:21.837190 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.837142    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47cae6de-3591-4321-87d6-51fa63e020bb-tls-certs\") pod \"47cae6de-3591-4321-87d6-51fa63e020bb\" (UID: \"47cae6de-3591-4321-87d6-51fa63e020bb\") "
Sep 17 18:53:21.839485 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.839454    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cae6de-3591-4321-87d6-51fa63e020bb-kube-api-access-8n7kw" (OuterVolumeSpecName: "kube-api-access-8n7kw") pod "47cae6de-3591-4321-87d6-51fa63e020bb" (UID: "47cae6de-3591-4321-87d6-51fa63e020bb"). InnerVolumeSpecName "kube-api-access-8n7kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:53:21.839577 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.839526    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47cae6de-3591-4321-87d6-51fa63e020bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47cae6de-3591-4321-87d6-51fa63e020bb" (UID: "47cae6de-3591-4321-87d6-51fa63e020bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:53:21.853203 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.853181    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb_47cae6de-3591-4321-87d6-51fa63e020bb/main/0.log"
Sep 17 18:53:21.853307 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.853220    2587 generic.go:356] "Generic (PLEG): container finished" podID="47cae6de-3591-4321-87d6-51fa63e020bb" containerID="9cab6ddc7cc9a69c44a042afa8e3c200790883c2be342f7a3b9dc19e9135966a" exitCode=1
Sep 17 18:53:21.853307 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.853282    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"
Sep 17 18:53:21.853374 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.853311    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb" event={"ID":"47cae6de-3591-4321-87d6-51fa63e020bb","Type":"ContainerDied","Data":"9cab6ddc7cc9a69c44a042afa8e3c200790883c2be342f7a3b9dc19e9135966a"}
Sep 17 18:53:21.853374 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.853362    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb" event={"ID":"47cae6de-3591-4321-87d6-51fa63e020bb","Type":"ContainerDied","Data":"4d90c855190a0175d224fae6876611ba370ce42bc32c7330e388a5d7e62adda6"}
Sep 17 18:53:21.853438 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.853385    2587 scope.go:117] "RemoveContainer" containerID="9cab6ddc7cc9a69c44a042afa8e3c200790883c2be342f7a3b9dc19e9135966a"
Sep 17 18:53:21.862361 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.862345    2587 scope.go:117] "RemoveContainer" containerID="9cab6ddc7cc9a69c44a042afa8e3c200790883c2be342f7a3b9dc19e9135966a"
Sep 17 18:53:21.862610 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:53:21.862594    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cab6ddc7cc9a69c44a042afa8e3c200790883c2be342f7a3b9dc19e9135966a\": container with ID starting with 9cab6ddc7cc9a69c44a042afa8e3c200790883c2be342f7a3b9dc19e9135966a not found: ID does not exist" containerID="9cab6ddc7cc9a69c44a042afa8e3c200790883c2be342f7a3b9dc19e9135966a"
Sep 17 18:53:21.862656 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.862621    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cab6ddc7cc9a69c44a042afa8e3c200790883c2be342f7a3b9dc19e9135966a"} err="failed to get container status \"9cab6ddc7cc9a69c44a042afa8e3c200790883c2be342f7a3b9dc19e9135966a\": rpc error: code = NotFound desc = could not find container \"9cab6ddc7cc9a69c44a042afa8e3c200790883c2be342f7a3b9dc19e9135966a\": container with ID starting with 9cab6ddc7cc9a69c44a042afa8e3c200790883c2be342f7a3b9dc19e9135966a not found: ID does not exist"
Sep 17 18:53:21.880168 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.876637    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"]
Sep 17 18:53:21.883324 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.883287    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc55gbtb"]
Sep 17 18:53:21.938601 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.938574    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n7kw\" (UniqueName: \"kubernetes.io/projected/47cae6de-3591-4321-87d6-51fa63e020bb-kube-api-access-8n7kw\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:53:21.938601 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:21.938600    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47cae6de-3591-4321-87d6-51fa63e020bb-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:53:22.257458 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:22.257427    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47cae6de-3591-4321-87d6-51fa63e020bb" path="/var/lib/kubelet/pods/47cae6de-3591-4321-87d6-51fa63e020bb/volumes"
Sep 17 18:53:59.396837 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:59.396755    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:59.493530 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:53:59.493498    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:06.004509 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:06.004471    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:08.151439 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:08.151409    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:08.251785 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:08.251758    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:10.190137 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:10.190106    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 18:54:10.190872 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:10.190856    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 18:54:11.597478 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:11.597441    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:11.680749 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:11.680702    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:11.900357 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:11.900280    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"]
Sep 17 18:54:11.900621 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:11.900609    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47cae6de-3591-4321-87d6-51fa63e020bb" containerName="main"
Sep 17 18:54:11.900663 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:11.900623    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="47cae6de-3591-4321-87d6-51fa63e020bb" containerName="main"
Sep 17 18:54:11.900701 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:11.900677    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="47cae6de-3591-4321-87d6-51fa63e020bb" containerName="main"
Sep 17 18:54:11.904686 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:11.904669    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:11.908974 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:11.908934    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 18:54:11.908974 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:11.908959    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 18:54:11.909141 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:11.908998    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 18:54:11.909327 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:11.909309    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-25p5g\""
Sep 17 18:54:11.917715 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:11.917689    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"]
Sep 17 18:54:11.927021 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:11.926990    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:12.050327 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.050290    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72t6\" (UniqueName: \"kubernetes.io/projected/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-kube-api-access-v72t6\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.050327 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.050334    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.050594 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.050417    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.050594 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.050472    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.050594 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.050563    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.050594 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.050590    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.151107 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.151003    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72t6\" (UniqueName: \"kubernetes.io/projected/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-kube-api-access-v72t6\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.151107 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.151049    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.151338 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.151108    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.151338 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.151133    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.151338 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.151276    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.151338 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.151312    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.151572 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.151550    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.151631 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.151569    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.151720 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.151695    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.153867 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.153842    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.154237 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.154217    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.160296 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.160270    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72t6\" (UniqueName: \"kubernetes.io/projected/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-kube-api-access-v72t6\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-m5rmr\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.195161 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.195115    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:12.214992 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.214956    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:12.492412 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:12.492380    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"]
Sep 17 18:54:12.492878 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:54:12.492853    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35dc913c_5e0f_4f39_bcde_73ddeae21e8d.slice/crio-cad63c62c1b72f3da8a3b1e58c22a4ed8296a78aae4bd46af13ebd6f63a408e3 WatchSource:0}: Error finding container cad63c62c1b72f3da8a3b1e58c22a4ed8296a78aae4bd46af13ebd6f63a408e3: Status 404 returned error can't find the container with id cad63c62c1b72f3da8a3b1e58c22a4ed8296a78aae4bd46af13ebd6f63a408e3
Sep 17 18:54:13.079235 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:13.079199    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr" event={"ID":"35dc913c-5e0f-4f39-bcde-73ddeae21e8d","Type":"ContainerStarted","Data":"cad63c62c1b72f3da8a3b1e58c22a4ed8296a78aae4bd46af13ebd6f63a408e3"}
Sep 17 18:54:14.085389 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:14.085348    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr" event={"ID":"35dc913c-5e0f-4f39-bcde-73ddeae21e8d","Type":"ContainerStarted","Data":"c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f"}
Sep 17 18:54:14.103927 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:14.103880    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr" podStartSLOduration=1.646485457 podStartE2EDuration="3.103855975s" totalImagesPullingTime="1.457370518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:14.103396809 +0000 UTC m=+905.982365568" watchObservedRunningTime="2026-09-17 18:54:14.103855975 +0000 UTC m=+905.982824734"
Sep 17 18:54:14.215851 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:14.215760    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:15.122743 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:15.122708    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:20.968386 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:20.968342    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:21.066052 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:21.066015    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:22.215529 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:22.215498    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:22.227960 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:22.227919    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:23.141854 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:23.141826    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:24.203384 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:24.203341    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:38.933536 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:38.933495    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:39.262299 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:39.262260    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:48.717603 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:48.717563    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:53.113099 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.113058    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:53.118587 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.118557    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"]
Sep 17 18:54:53.128340 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:54:53.128293    2587 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:54:53.128340 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:54:53.128340 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 18:54:53.128340 ip-10-0-140-194 kubenswrapper[2587]:  > containerID="c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f" cmd=["/bin/sleep","15"]
Sep 17 18:54:53.128671 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.128370    2587 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr" message="" err=<
Sep 17 18:54:53.128671 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:54:53.128671 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 18:54:53.128671 ip-10-0-140-194 kubenswrapper[2587]:  >
Sep 17 18:54:53.128671 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:54:53.128389    2587 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:54:53.128671 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:54:53.128671 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 18:54:53.128671 ip-10-0-140-194 kubenswrapper[2587]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr" podUID="35dc913c-5e0f-4f39-bcde-73ddeae21e8d" containerName="main" containerID="cri-o://c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f"
Sep 17 18:54:53.128671 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.128438    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr" podUID="35dc913c-5e0f-4f39-bcde-73ddeae21e8d" containerName="main" containerID="cri-o://c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f" gracePeriod=60
Sep 17 18:54:53.244414 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.244387    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:53.485107 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.485078    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:53.592671 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.592637    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-model-cache\") pod \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") "
Sep 17 18:54:53.592882 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.592698    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-tmp-dir\") pod \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") "
Sep 17 18:54:53.592882 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.592726    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-dshm\") pod \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") "
Sep 17 18:54:53.592882 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.592743    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-tls-certs\") pod \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") "
Sep 17 18:54:53.592882 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.592789    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v72t6\" (UniqueName: \"kubernetes.io/projected/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-kube-api-access-v72t6\") pod \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") "
Sep 17 18:54:53.592882 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.592817    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-home\") pod \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\" (UID: \"35dc913c-5e0f-4f39-bcde-73ddeae21e8d\") "
Sep 17 18:54:53.593171 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.592921    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "35dc913c-5e0f-4f39-bcde-73ddeae21e8d" (UID: "35dc913c-5e0f-4f39-bcde-73ddeae21e8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:53.593171 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.593037    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:54:53.593171 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.593099    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35dc913c-5e0f-4f39-bcde-73ddeae21e8d" (UID: "35dc913c-5e0f-4f39-bcde-73ddeae21e8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:53.593171 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.593132    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-home" (OuterVolumeSpecName: "home") pod "35dc913c-5e0f-4f39-bcde-73ddeae21e8d" (UID: "35dc913c-5e0f-4f39-bcde-73ddeae21e8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:53.595342 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.595310    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-dshm" (OuterVolumeSpecName: "dshm") pod "35dc913c-5e0f-4f39-bcde-73ddeae21e8d" (UID: "35dc913c-5e0f-4f39-bcde-73ddeae21e8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:53.595342 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.595308    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-kube-api-access-v72t6" (OuterVolumeSpecName: "kube-api-access-v72t6") pod "35dc913c-5e0f-4f39-bcde-73ddeae21e8d" (UID: "35dc913c-5e0f-4f39-bcde-73ddeae21e8d"). InnerVolumeSpecName "kube-api-access-v72t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:53.595472 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.595344    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35dc913c-5e0f-4f39-bcde-73ddeae21e8d" (UID: "35dc913c-5e0f-4f39-bcde-73ddeae21e8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:53.694091 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.694000    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:54:53.694091 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.694030    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:54:53.694091 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.694038    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:54:53.694091 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.694046    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v72t6\" (UniqueName: \"kubernetes.io/projected/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-kube-api-access-v72t6\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:54:53.694091 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:53.694056    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35dc913c-5e0f-4f39-bcde-73ddeae21e8d-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:54:54.272186 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:54.272158    2587 generic.go:356] "Generic (PLEG): container finished" podID="35dc913c-5e0f-4f39-bcde-73ddeae21e8d" containerID="c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f" exitCode=0
Sep 17 18:54:54.272620 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:54.272201    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"
Sep 17 18:54:54.272620 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:54.272234    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr" event={"ID":"35dc913c-5e0f-4f39-bcde-73ddeae21e8d","Type":"ContainerDied","Data":"c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f"}
Sep 17 18:54:54.272620 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:54.272301    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr" event={"ID":"35dc913c-5e0f-4f39-bcde-73ddeae21e8d","Type":"ContainerDied","Data":"cad63c62c1b72f3da8a3b1e58c22a4ed8296a78aae4bd46af13ebd6f63a408e3"}
Sep 17 18:54:54.272620 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:54.272336    2587 scope.go:117] "RemoveContainer" containerID="c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f"
Sep 17 18:54:54.281525 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:54.281507    2587 scope.go:117] "RemoveContainer" containerID="c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f"
Sep 17 18:54:54.281837 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:54:54.281819    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f\": container with ID starting with c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f not found: ID does not exist" containerID="c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f"
Sep 17 18:54:54.281880 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:54.281847    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f"} err="failed to get container status \"c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f\": rpc error: code = NotFound desc = could not find container \"c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f\": container with ID starting with c1af254febf0aa8abb9a8ca609263c6f304e54827aff7eed1ca6641a2c84a10f not found: ID does not exist"
Sep 17 18:54:54.290707 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:54.290678    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"]
Sep 17 18:54:54.294148 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:54.294123    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-m5rmr"]
Sep 17 18:54:56.258124 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:54:56.258085    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35dc913c-5e0f-4f39-bcde-73ddeae21e8d" path="/var/lib/kubelet/pods/35dc913c-5e0f-4f39-bcde-73ddeae21e8d/volumes"
Sep 17 18:55:18.327821 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:18.327772    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:18.426574 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:18.426536    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:41.897892 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:41.897854    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:42.083488 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:42.083451    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:48.791881 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.791848    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"]
Sep 17 18:55:48.792472 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.792331    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35dc913c-5e0f-4f39-bcde-73ddeae21e8d" containerName="main"
Sep 17 18:55:48.792472 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.792350    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="35dc913c-5e0f-4f39-bcde-73ddeae21e8d" containerName="main"
Sep 17 18:55:48.792472 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.792435    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="35dc913c-5e0f-4f39-bcde-73ddeae21e8d" containerName="main"
Sep 17 18:55:48.795484 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.795464    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.803413 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.803388    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 18:55:48.803520 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.803389    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 18:55:48.803520 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.803391    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-6rxm2\""
Sep 17 18:55:48.803520 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.803389    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 18:55:48.806011 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.805996    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-k76mf\""
Sep 17 18:55:48.819927 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.819903    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"]
Sep 17 18:55:48.822149 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.822126    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.822203 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.822179    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf28c\" (UniqueName: \"kubernetes.io/projected/a9e807ab-cdc2-48a6-9040-d2020bbacd34-kube-api-access-gf28c\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.822239 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.822223    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.822287 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.822272    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.822349 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.822336    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9e807ab-cdc2-48a6-9040-d2020bbacd34-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.822384 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.822361    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.871322 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.871291    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:48.923185 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.923163    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9e807ab-cdc2-48a6-9040-d2020bbacd34-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.923272 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.923189    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.923272 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.923222    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.923272 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.923252    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf28c\" (UniqueName: \"kubernetes.io/projected/a9e807ab-cdc2-48a6-9040-d2020bbacd34-kube-api-access-gf28c\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.923416 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.923282    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.923416 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.923311    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.923680 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.923647    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.923680 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.923662    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.923799 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.923686    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.925672 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.925652    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.925958 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.925923    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9e807ab-cdc2-48a6-9040-d2020bbacd34-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:48.933461 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:48.933443    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf28c\" (UniqueName: \"kubernetes.io/projected/a9e807ab-cdc2-48a6-9040-d2020bbacd34-kube-api-access-gf28c\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:49.104748 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:49.104655    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:49.232557 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:49.232521    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:49.389250 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:49.389226    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"]
Sep 17 18:55:49.390707 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:55:49.390674    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e807ab_cdc2_48a6_9040_d2020bbacd34.slice/crio-95fcf7c90c8c414a0ba2b3a4cd0588836fe6c97d33dc6013fbda497dec9c25ff WatchSource:0}: Error finding container 95fcf7c90c8c414a0ba2b3a4cd0588836fe6c97d33dc6013fbda497dec9c25ff: Status 404 returned error can't find the container with id 95fcf7c90c8c414a0ba2b3a4cd0588836fe6c97d33dc6013fbda497dec9c25ff
Sep 17 18:55:49.493763 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:49.493728    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7" event={"ID":"a9e807ab-cdc2-48a6-9040-d2020bbacd34","Type":"ContainerStarted","Data":"95fcf7c90c8c414a0ba2b3a4cd0588836fe6c97d33dc6013fbda497dec9c25ff"}
Sep 17 18:55:49.706866 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:49.706779    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:51.505799 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:51.505764    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7" event={"ID":"a9e807ab-cdc2-48a6-9040-d2020bbacd34","Type":"ContainerStarted","Data":"ec5d549f13cb44675206f4edd9d46330689155f67164b89e444319f667932499"}
Sep 17 18:55:51.506302 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:51.505843    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:52.513203 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:52.513162    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7" event={"ID":"a9e807ab-cdc2-48a6-9040-d2020bbacd34","Type":"ContainerStarted","Data":"b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92"}
Sep 17 18:55:52.552418 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:52.552367    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7" podStartSLOduration=3.048561445 podStartE2EDuration="4.55234832s" totalImagesPullingTime="1.503786875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:52.546901301 +0000 UTC m=+1004.425870057" watchObservedRunningTime="2026-09-17 18:55:52.55234832 +0000 UTC m=+1004.431317125"
Sep 17 18:55:53.105709 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:53.105660    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:55.533987 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:55.533922    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:55.627777 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:55.627737    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:59.104849 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:59.104812    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:59.117393 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:59.117359    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:55:59.555277 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:55:59.555250    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:56:00.560889 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:00.560861    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:56:01.412519 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:01.412482    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:08.389520 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:08.389482    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:08.494805 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:08.494770    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:08.595803 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:08.595758    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:08.879677 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:08.879639    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:08.996330 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:08.996287    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:12.527369 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:12.527333    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:14.762637 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:14.762603    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:20.608578 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:20.608541    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:20.709698 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:20.709663    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:21.444228 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:21.444198    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:21.773564 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:21.773526    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:23.627264 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:23.627205    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:23.736035 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:23.735998    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:23.835857 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:23.835819    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:23.962471 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:23.962425    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:24.064047 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:24.064012    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:26.866107 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:26.866076    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:28.708491 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:28.708464    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:29.689640 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:29.689606    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:33.970917 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:33.970882    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:34.803861 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:34.803824    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:39.198958 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:39.198907    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:40.180681 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:40.180646    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:40.911889 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:40.911858    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:41.009905 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:41.009866    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:42.520462 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:42.520425    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:42.715372 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:42.715343    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:44.649743 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:44.649706    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:47.584268 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:47.584230    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:53.032618 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:53.032526    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:54.090084 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:54.090053    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:57.817043 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.817011    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"]
Sep 17 18:56:57.820388 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.820365    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:57.824115 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.824087    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:56:57.824115 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.824105    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 17 18:56:57.824363 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.824339    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 18:56:57.824433 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.824405    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 18:56:57.824535 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.824468    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 18:56:57.824535 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.824524    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-wftxp\""
Sep 17 18:56:57.835774 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.835732    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"]
Sep 17 18:56:57.851736 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.851699    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:57.900871 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.900843    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-home\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:57.901045 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.900874    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwdp4\" (UniqueName: \"kubernetes.io/projected/573d2e30-b1fb-45c9-87fe-623417f1c32b-kube-api-access-mwdp4\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:57.901045 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.900897    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-kserve-provision-location\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:57.901045 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.900914    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573d2e30-b1fb-45c9-87fe-623417f1c32b-tls-certs\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:57.901045 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.900963    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-tmp-dir\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:57.901045 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.901004    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/573d2e30-b1fb-45c9-87fe-623417f1c32b-cabundle-cert\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:57.901045 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.901019    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-dshm\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:57.901251 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:57.901096    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-model-cache\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.002262 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.002211    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573d2e30-b1fb-45c9-87fe-623417f1c32b-tls-certs\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.002262 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.002263    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-tmp-dir\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.002522 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.002300    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/573d2e30-b1fb-45c9-87fe-623417f1c32b-cabundle-cert\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.002522 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.002319    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-dshm\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.002522 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.002347    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-model-cache\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.002522 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.002385    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-home\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.002522 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.002503    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwdp4\" (UniqueName: \"kubernetes.io/projected/573d2e30-b1fb-45c9-87fe-623417f1c32b-kube-api-access-mwdp4\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.002724 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.002543    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-kserve-provision-location\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.003077 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.003020    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-model-cache\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.003077 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.003056    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-home\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.003243 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.003117    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-kserve-provision-location\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.003243 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.003151    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-tmp-dir\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.003372 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.003351    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/573d2e30-b1fb-45c9-87fe-623417f1c32b-cabundle-cert\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.004830 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.004805    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-dshm\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.005111 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.005094    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573d2e30-b1fb-45c9-87fe-623417f1c32b-tls-certs\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.012683 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.012661    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwdp4\" (UniqueName: \"kubernetes.io/projected/573d2e30-b1fb-45c9-87fe-623417f1c32b-kube-api-access-mwdp4\") pod \"auth-enabled-test-kserve-75dd8cfd66-vt89h\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.090275 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.090175    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:58.137007 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.136975    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:56:58.422022 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:56:58.421994    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573d2e30_b1fb_45c9_87fe_623417f1c32b.slice/crio-4b756427a81e3590e8a7a368bc678dfa257014791f6cdb426c2a7704fc9f3b7d WatchSource:0}: Error finding container 4b756427a81e3590e8a7a368bc678dfa257014791f6cdb426c2a7704fc9f3b7d: Status 404 returned error can't find the container with id 4b756427a81e3590e8a7a368bc678dfa257014791f6cdb426c2a7704fc9f3b7d
Sep 17 18:56:58.422376 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.422354    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"]
Sep 17 18:56:58.808909 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.808871    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" event={"ID":"573d2e30-b1fb-45c9-87fe-623417f1c32b","Type":"ContainerStarted","Data":"611e58be21b580d3e3f769681fa2688e6fc51044b2db5dcfe3c9ba16ef583c66"}
Sep 17 18:56:58.809118 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:58.808929    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" event={"ID":"573d2e30-b1fb-45c9-87fe-623417f1c32b","Type":"ContainerStarted","Data":"4b756427a81e3590e8a7a368bc678dfa257014791f6cdb426c2a7704fc9f3b7d"}
Sep 17 18:56:59.449424 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:56:59.449392    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:00.821597 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:00.821569    2587 generic.go:356] "Generic (PLEG): container finished" podID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerID="611e58be21b580d3e3f769681fa2688e6fc51044b2db5dcfe3c9ba16ef583c66" exitCode=0
Sep 17 18:57:00.821963 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:00.821646    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" event={"ID":"573d2e30-b1fb-45c9-87fe-623417f1c32b","Type":"ContainerDied","Data":"611e58be21b580d3e3f769681fa2688e6fc51044b2db5dcfe3c9ba16ef583c66"}
Sep 17 18:57:06.819784 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:06.819736    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:09.363506 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:09.363465    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:09.464286 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:09.464246    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:28.982866 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:28.982825    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" event={"ID":"573d2e30-b1fb-45c9-87fe-623417f1c32b","Type":"ContainerStarted","Data":"a72dfac225d7b71c494b575948e288795d27585d7de9a9272743149ed47aa728"}
Sep 17 18:57:29.013225 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:29.013167    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" podStartSLOduration=2.506073254 podStartE2EDuration="32.013153508s" totalImagesPullingTime="27.316040432s" totalInitContainerRuntime="2.191039822s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:57:29.01070738 +0000 UTC m=+1100.889676150" watchObservedRunningTime="2026-09-17 18:57:29.013153508 +0000 UTC m=+1100.892122264"
Sep 17 18:57:29.137575 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:29.137515    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:57:36.606717 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:36.606678    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:36.635100 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:36.635063    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"]
Sep 17 18:57:36.644191 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:57:36.644121    2587 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:57:36.644191 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:57:36.644191 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 18:57:36.644191 ip-10-0-140-194 kubenswrapper[2587]:  > containerID="b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92" cmd=["/bin/sleep","15"]
Sep 17 18:57:36.644481 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:36.644217    2587 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7" message="" err=<
Sep 17 18:57:36.644481 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:57:36.644481 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 18:57:36.644481 ip-10-0-140-194 kubenswrapper[2587]:  >
Sep 17 18:57:36.644481 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:57:36.644243    2587 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:57:36.644481 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:57:36.644481 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 18:57:36.644481 ip-10-0-140-194 kubenswrapper[2587]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7" podUID="a9e807ab-cdc2-48a6-9040-d2020bbacd34" containerName="main" containerID="cri-o://b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92"
Sep 17 18:57:36.644481 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:36.644302    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7" podUID="a9e807ab-cdc2-48a6-9040-d2020bbacd34" containerName="main" containerID="cri-o://b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92" gracePeriod=60
Sep 17 18:57:36.801773 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:36.801739    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:37.028119 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.028083    2587 generic.go:356] "Generic (PLEG): container finished" podID="a9e807ab-cdc2-48a6-9040-d2020bbacd34" containerID="b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92" exitCode=0
Sep 17 18:57:37.028305 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.028152    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7" event={"ID":"a9e807ab-cdc2-48a6-9040-d2020bbacd34","Type":"ContainerDied","Data":"b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92"}
Sep 17 18:57:37.426298 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.426264    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7" podUID="a9e807ab-cdc2-48a6-9040-d2020bbacd34" containerName="llm-d-routing-sidecar" containerID="cri-o://ec5d549f13cb44675206f4edd9d46330689155f67164b89e444319f667932499" gracePeriod=60
Sep 17 18:57:37.545006 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.544965    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7" podUID="a9e807ab-cdc2-48a6-9040-d2020bbacd34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8001/health\": dial tcp 10.132.0.31:8001: connect: connection refused"
Sep 17 18:57:37.742192 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.742156    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:57:37.871899 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.871865    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf28c\" (UniqueName: \"kubernetes.io/projected/a9e807ab-cdc2-48a6-9040-d2020bbacd34-kube-api-access-gf28c\") pod \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") "
Sep 17 18:57:37.872101 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.871970    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-home\") pod \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") "
Sep 17 18:57:37.872101 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.872047    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9e807ab-cdc2-48a6-9040-d2020bbacd34-tls-certs\") pod \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") "
Sep 17 18:57:37.872101 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.872074    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-dshm\") pod \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") "
Sep 17 18:57:37.872101 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.872094    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-tmp-dir\") pod \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") "
Sep 17 18:57:37.872326 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.872128    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-model-cache\") pod \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\" (UID: \"a9e807ab-cdc2-48a6-9040-d2020bbacd34\") "
Sep 17 18:57:37.872326 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.872267    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-home" (OuterVolumeSpecName: "home") pod "a9e807ab-cdc2-48a6-9040-d2020bbacd34" (UID: "a9e807ab-cdc2-48a6-9040-d2020bbacd34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:37.872437 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.872397    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9e807ab-cdc2-48a6-9040-d2020bbacd34" (UID: "a9e807ab-cdc2-48a6-9040-d2020bbacd34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:37.872437 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.872425    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.872533 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.872436    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9e807ab-cdc2-48a6-9040-d2020bbacd34" (UID: "a9e807ab-cdc2-48a6-9040-d2020bbacd34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:37.874488 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.874458    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e807ab-cdc2-48a6-9040-d2020bbacd34-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9e807ab-cdc2-48a6-9040-d2020bbacd34" (UID: "a9e807ab-cdc2-48a6-9040-d2020bbacd34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:57:37.874600 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.874523    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-dshm" (OuterVolumeSpecName: "dshm") pod "a9e807ab-cdc2-48a6-9040-d2020bbacd34" (UID: "a9e807ab-cdc2-48a6-9040-d2020bbacd34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:37.874600 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.874532    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e807ab-cdc2-48a6-9040-d2020bbacd34-kube-api-access-gf28c" (OuterVolumeSpecName: "kube-api-access-gf28c") pod "a9e807ab-cdc2-48a6-9040-d2020bbacd34" (UID: "a9e807ab-cdc2-48a6-9040-d2020bbacd34"). InnerVolumeSpecName "kube-api-access-gf28c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:57:37.973905 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.973812    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf28c\" (UniqueName: \"kubernetes.io/projected/a9e807ab-cdc2-48a6-9040-d2020bbacd34-kube-api-access-gf28c\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.973905 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.973890    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9e807ab-cdc2-48a6-9040-d2020bbacd34-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.974111 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.973917    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.974111 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.973926    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.974111 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:37.973934    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9e807ab-cdc2-48a6-9040-d2020bbacd34-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:57:38.034321 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.034284    2587 generic.go:356] "Generic (PLEG): container finished" podID="a9e807ab-cdc2-48a6-9040-d2020bbacd34" containerID="ec5d549f13cb44675206f4edd9d46330689155f67164b89e444319f667932499" exitCode=0
Sep 17 18:57:38.034456 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.034334    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"
Sep 17 18:57:38.034456 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.034369    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7" event={"ID":"a9e807ab-cdc2-48a6-9040-d2020bbacd34","Type":"ContainerDied","Data":"ec5d549f13cb44675206f4edd9d46330689155f67164b89e444319f667932499"}
Sep 17 18:57:38.034456 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.034425    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7" event={"ID":"a9e807ab-cdc2-48a6-9040-d2020bbacd34","Type":"ContainerDied","Data":"95fcf7c90c8c414a0ba2b3a4cd0588836fe6c97d33dc6013fbda497dec9c25ff"}
Sep 17 18:57:38.034456 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.034448    2587 scope.go:117] "RemoveContainer" containerID="b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92"
Sep 17 18:57:38.044281 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.044254    2587 scope.go:117] "RemoveContainer" containerID="ec5d549f13cb44675206f4edd9d46330689155f67164b89e444319f667932499"
Sep 17 18:57:38.053291 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.053268    2587 scope.go:117] "RemoveContainer" containerID="b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92"
Sep 17 18:57:38.053590 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:57:38.053571    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92\": container with ID starting with b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92 not found: ID does not exist" containerID="b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92"
Sep 17 18:57:38.053672 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.053596    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92"} err="failed to get container status \"b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92\": rpc error: code = NotFound desc = could not find container \"b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92\": container with ID starting with b74592d5443fba1f352d8f2375471f2db5cc10e80ce761b2a84321de64c35c92 not found: ID does not exist"
Sep 17 18:57:38.053672 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.053613    2587 scope.go:117] "RemoveContainer" containerID="ec5d549f13cb44675206f4edd9d46330689155f67164b89e444319f667932499"
Sep 17 18:57:38.053902 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:57:38.053888    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5d549f13cb44675206f4edd9d46330689155f67164b89e444319f667932499\": container with ID starting with ec5d549f13cb44675206f4edd9d46330689155f67164b89e444319f667932499 not found: ID does not exist" containerID="ec5d549f13cb44675206f4edd9d46330689155f67164b89e444319f667932499"
Sep 17 18:57:38.053956 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.053906    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5d549f13cb44675206f4edd9d46330689155f67164b89e444319f667932499"} err="failed to get container status \"ec5d549f13cb44675206f4edd9d46330689155f67164b89e444319f667932499\": rpc error: code = NotFound desc = could not find container \"ec5d549f13cb44675206f4edd9d46330689155f67164b89e444319f667932499\": container with ID starting with ec5d549f13cb44675206f4edd9d46330689155f67164b89e444319f667932499 not found: ID does not exist"
Sep 17 18:57:38.057562 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.057537    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"]
Sep 17 18:57:38.061847 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.061821    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-xlsk7"]
Sep 17 18:57:38.137134 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.137091    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:57:38.138928 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.138896    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 17 18:57:38.257922 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:38.257888    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9e807ab-cdc2-48a6-9040-d2020bbacd34" path="/var/lib/kubelet/pods/a9e807ab-cdc2-48a6-9040-d2020bbacd34/volumes"
Sep 17 18:57:48.137729 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:48.137686    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 17 18:57:58.138255 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:57:58.138209    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 17 18:58:08.138076 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:08.138031    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 17 18:58:18.138223 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:18.138171    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 17 18:58:26.649868 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:26.649773    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:26.756028 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:26.755987    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:27.891679 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:27.891626    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:27.991266 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:27.991232    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:28.137865 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:28.137828    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 17 18:58:33.197760 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:33.197717    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:38.137912 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:38.137863    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 17 18:58:39.154888 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:39.154842    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:39.252989 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:39.252948    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:45.961096 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:45.961056    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"]
Sep 17 18:58:45.961524 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:45.961507    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e807ab-cdc2-48a6-9040-d2020bbacd34" containerName="main"
Sep 17 18:58:45.961572 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:45.961528    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e807ab-cdc2-48a6-9040-d2020bbacd34" containerName="main"
Sep 17 18:58:45.961572 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:45.961561    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e807ab-cdc2-48a6-9040-d2020bbacd34" containerName="llm-d-routing-sidecar"
Sep 17 18:58:45.961633 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:45.961570    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e807ab-cdc2-48a6-9040-d2020bbacd34" containerName="llm-d-routing-sidecar"
Sep 17 18:58:45.961672 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:45.961660    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e807ab-cdc2-48a6-9040-d2020bbacd34" containerName="llm-d-routing-sidecar"
Sep 17 18:58:45.961710 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:45.961675    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e807ab-cdc2-48a6-9040-d2020bbacd34" containerName="main"
Sep 17 18:58:45.964880 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:45.964857    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:45.970691 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:45.970665    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 18:58:45.970843 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:45.970692    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 18:58:45.970843 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:45.970667    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-t76vb\""
Sep 17 18:58:45.970843 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:45.970665    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 18:58:45.990032 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:45.989999    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"]
Sep 17 18:58:46.020503 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.020463    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:46.075827 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.075791    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.075827 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.075832    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.076074 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.075865    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.076074 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.075927    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.076074 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.075980    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e07439eb-40ff-4205-b26d-9c909153f93a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.076074 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.076001    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdj44\" (UniqueName: \"kubernetes.io/projected/e07439eb-40ff-4205-b26d-9c909153f93a-kube-api-access-kdj44\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.176537 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.176473    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e07439eb-40ff-4205-b26d-9c909153f93a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.176537 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.176524    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdj44\" (UniqueName: \"kubernetes.io/projected/e07439eb-40ff-4205-b26d-9c909153f93a-kube-api-access-kdj44\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.176847 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.176568    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.176847 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.176591    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.176847 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.176620    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.176847 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.176640    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.177204 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.177179    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.177204 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.177193    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.177341 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.177222    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.179643 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.179619    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.179888 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.179869    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e07439eb-40ff-4205-b26d-9c909153f93a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.193383 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.193341    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdj44\" (UniqueName: \"kubernetes.io/projected/e07439eb-40ff-4205-b26d-9c909153f93a-kube-api-access-kdj44\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.276589 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.276547    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:46.333131 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.332955    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:46.557239 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:46.557196    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"]
Sep 17 18:58:46.558799 ip-10-0-140-194 kubenswrapper[2587]: W0917 18:58:46.558775    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07439eb_40ff_4205_b26d_9c909153f93a.slice/crio-732cec03aa6817c7429f0fff1840958023ea6146a3a6571107c8fa5672771813 WatchSource:0}: Error finding container 732cec03aa6817c7429f0fff1840958023ea6146a3a6571107c8fa5672771813: Status 404 returned error can't find the container with id 732cec03aa6817c7429f0fff1840958023ea6146a3a6571107c8fa5672771813
Sep 17 18:58:47.372498 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:47.372461    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm" event={"ID":"e07439eb-40ff-4205-b26d-9c909153f93a","Type":"ContainerStarted","Data":"8d708475b53427ef9306956910dd6a9b9d5f6abc1d2b7955bdb620b9c90ac072"}
Sep 17 18:58:47.373036 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:47.372520    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm" event={"ID":"e07439eb-40ff-4205-b26d-9c909153f93a","Type":"ContainerStarted","Data":"732cec03aa6817c7429f0fff1840958023ea6146a3a6571107c8fa5672771813"}
Sep 17 18:58:47.400270 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:47.400211    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm" podStartSLOduration=2.400193581 podStartE2EDuration="2.400193581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:58:47.398921206 +0000 UTC m=+1179.277889964" watchObservedRunningTime="2026-09-17 18:58:47.400193581 +0000 UTC m=+1179.279162338"
Sep 17 18:58:48.138308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:48.138262    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 17 18:58:48.276860 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:48.276818    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:56.277065 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:56.277021    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:56.289910 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:56.289871    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:56.431535 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:56.431503    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:58:58.138070 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:58.138019    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 17 18:58:58.314975 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:58:58.314923    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:08.137388 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:08.137340    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 17 18:59:10.219025 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:10.218980    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 18:59:10.219439 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:10.219360    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 18:59:18.137716 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:18.137669    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 17 18:59:21.310033 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:21.309994    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"]
Sep 17 18:59:21.319708 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:59:21.319672    2587 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:59:21.319708 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:59:21.319708 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 18:59:21.319708 ip-10-0-140-194 kubenswrapper[2587]:  > containerID="8d708475b53427ef9306956910dd6a9b9d5f6abc1d2b7955bdb620b9c90ac072" cmd=["/bin/sleep","15"]
Sep 17 18:59:21.320057 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:21.319731    2587 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm" message="" err=<
Sep 17 18:59:21.320057 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:59:21.320057 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 18:59:21.320057 ip-10-0-140-194 kubenswrapper[2587]:  >
Sep 17 18:59:21.320057 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:59:21.319748    2587 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:59:21.320057 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:59:21.320057 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 18:59:21.320057 ip-10-0-140-194 kubenswrapper[2587]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm" podUID="e07439eb-40ff-4205-b26d-9c909153f93a" containerName="main" containerID="cri-o://8d708475b53427ef9306956910dd6a9b9d5f6abc1d2b7955bdb620b9c90ac072"
Sep 17 18:59:21.320057 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:21.319795    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm" podUID="e07439eb-40ff-4205-b26d-9c909153f93a" containerName="main" containerID="cri-o://8d708475b53427ef9306956910dd6a9b9d5f6abc1d2b7955bdb620b9c90ac072" gracePeriod=60
Sep 17 18:59:21.338288 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:21.338250    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:21.417504 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:21.417464    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm" podUID="e07439eb-40ff-4205-b26d-9c909153f93a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Sep 17 18:59:21.428324 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:21.428290    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:21.540094 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:21.540062    2587 generic.go:356] "Generic (PLEG): container finished" podID="e07439eb-40ff-4205-b26d-9c909153f93a" containerID="8d708475b53427ef9306956910dd6a9b9d5f6abc1d2b7955bdb620b9c90ac072" exitCode=0
Sep 17 18:59:21.540265 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:21.540137    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm" event={"ID":"e07439eb-40ff-4205-b26d-9c909153f93a","Type":"ContainerDied","Data":"8d708475b53427ef9306956910dd6a9b9d5f6abc1d2b7955bdb620b9c90ac072"}
Sep 17 18:59:22.068990 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.068962    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:59:22.210755 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.210659    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdj44\" (UniqueName: \"kubernetes.io/projected/e07439eb-40ff-4205-b26d-9c909153f93a-kube-api-access-kdj44\") pod \"e07439eb-40ff-4205-b26d-9c909153f93a\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") "
Sep 17 18:59:22.210755 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.210721    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-home\") pod \"e07439eb-40ff-4205-b26d-9c909153f93a\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") "
Sep 17 18:59:22.211107 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.210764    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-dshm\") pod \"e07439eb-40ff-4205-b26d-9c909153f93a\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") "
Sep 17 18:59:22.211107 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.210780    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e07439eb-40ff-4205-b26d-9c909153f93a-tls-certs\") pod \"e07439eb-40ff-4205-b26d-9c909153f93a\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") "
Sep 17 18:59:22.211107 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.210805    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-tmp-dir\") pod \"e07439eb-40ff-4205-b26d-9c909153f93a\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") "
Sep 17 18:59:22.211107 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.210895    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-model-cache\") pod \"e07439eb-40ff-4205-b26d-9c909153f93a\" (UID: \"e07439eb-40ff-4205-b26d-9c909153f93a\") "
Sep 17 18:59:22.211339 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.211132    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-home" (OuterVolumeSpecName: "home") pod "e07439eb-40ff-4205-b26d-9c909153f93a" (UID: "e07439eb-40ff-4205-b26d-9c909153f93a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:22.211339 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.211178    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e07439eb-40ff-4205-b26d-9c909153f93a" (UID: "e07439eb-40ff-4205-b26d-9c909153f93a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:22.211339 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.211224    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e07439eb-40ff-4205-b26d-9c909153f93a" (UID: "e07439eb-40ff-4205-b26d-9c909153f93a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:22.213229 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.213207    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e07439eb-40ff-4205-b26d-9c909153f93a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e07439eb-40ff-4205-b26d-9c909153f93a" (UID: "e07439eb-40ff-4205-b26d-9c909153f93a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:59:22.213322 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.213271    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07439eb-40ff-4205-b26d-9c909153f93a-kube-api-access-kdj44" (OuterVolumeSpecName: "kube-api-access-kdj44") pod "e07439eb-40ff-4205-b26d-9c909153f93a" (UID: "e07439eb-40ff-4205-b26d-9c909153f93a"). InnerVolumeSpecName "kube-api-access-kdj44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:59:22.213322 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.213280    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-dshm" (OuterVolumeSpecName: "dshm") pod "e07439eb-40ff-4205-b26d-9c909153f93a" (UID: "e07439eb-40ff-4205-b26d-9c909153f93a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:22.312248 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.312211    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:59:22.312248 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.312241    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e07439eb-40ff-4205-b26d-9c909153f93a-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:59:22.312248 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.312250    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:59:22.312248 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.312260    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:59:22.312737 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.312269    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdj44\" (UniqueName: \"kubernetes.io/projected/e07439eb-40ff-4205-b26d-9c909153f93a-kube-api-access-kdj44\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:59:22.312737 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.312277    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e07439eb-40ff-4205-b26d-9c909153f93a-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:59:22.546177 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.546142    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm" event={"ID":"e07439eb-40ff-4205-b26d-9c909153f93a","Type":"ContainerDied","Data":"732cec03aa6817c7429f0fff1840958023ea6146a3a6571107c8fa5672771813"}
Sep 17 18:59:22.546177 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.546176    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"
Sep 17 18:59:22.546401 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.546206    2587 scope.go:117] "RemoveContainer" containerID="8d708475b53427ef9306956910dd6a9b9d5f6abc1d2b7955bdb620b9c90ac072"
Sep 17 18:59:22.573747 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.573715    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"]
Sep 17 18:59:22.579644 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:22.579611    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444495bm"]
Sep 17 18:59:24.258281 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:24.258245    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e07439eb-40ff-4205-b26d-9c909153f93a" path="/var/lib/kubelet/pods/e07439eb-40ff-4205-b26d-9c909153f93a/volumes"
Sep 17 18:59:28.147505 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:28.147470    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:59:28.155407 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:28.155372    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:59:33.786110 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:33.786066    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:33.871812 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:33.871775    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"]
Sep 17 18:59:33.962511 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:33.962476    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:48.895085 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:48.895039    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main" containerID="cri-o://a72dfac225d7b71c494b575948e288795d27585d7de9a9272743149ed47aa728" gracePeriod=45
Sep 17 18:59:49.148902 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.148796    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 17 18:59:49.651073 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.651046    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:59:49.670519 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.670486    2587 generic.go:356] "Generic (PLEG): container finished" podID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerID="a72dfac225d7b71c494b575948e288795d27585d7de9a9272743149ed47aa728" exitCode=0
Sep 17 18:59:49.670684 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.670542    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"
Sep 17 18:59:49.670684 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.670567    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" event={"ID":"573d2e30-b1fb-45c9-87fe-623417f1c32b","Type":"ContainerDied","Data":"a72dfac225d7b71c494b575948e288795d27585d7de9a9272743149ed47aa728"}
Sep 17 18:59:49.670684 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.670627    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h" event={"ID":"573d2e30-b1fb-45c9-87fe-623417f1c32b","Type":"ContainerDied","Data":"4b756427a81e3590e8a7a368bc678dfa257014791f6cdb426c2a7704fc9f3b7d"}
Sep 17 18:59:49.670684 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.670654    2587 scope.go:117] "RemoveContainer" containerID="a72dfac225d7b71c494b575948e288795d27585d7de9a9272743149ed47aa728"
Sep 17 18:59:49.679259 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.679239    2587 scope.go:117] "RemoveContainer" containerID="611e58be21b580d3e3f769681fa2688e6fc51044b2db5dcfe3c9ba16ef583c66"
Sep 17 18:59:49.688921 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.688900    2587 scope.go:117] "RemoveContainer" containerID="a72dfac225d7b71c494b575948e288795d27585d7de9a9272743149ed47aa728"
Sep 17 18:59:49.689344 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:59:49.689323    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72dfac225d7b71c494b575948e288795d27585d7de9a9272743149ed47aa728\": container with ID starting with a72dfac225d7b71c494b575948e288795d27585d7de9a9272743149ed47aa728 not found: ID does not exist" containerID="a72dfac225d7b71c494b575948e288795d27585d7de9a9272743149ed47aa728"
Sep 17 18:59:49.689433 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.689353    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72dfac225d7b71c494b575948e288795d27585d7de9a9272743149ed47aa728"} err="failed to get container status \"a72dfac225d7b71c494b575948e288795d27585d7de9a9272743149ed47aa728\": rpc error: code = NotFound desc = could not find container \"a72dfac225d7b71c494b575948e288795d27585d7de9a9272743149ed47aa728\": container with ID starting with a72dfac225d7b71c494b575948e288795d27585d7de9a9272743149ed47aa728 not found: ID does not exist"
Sep 17 18:59:49.689433 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.689372    2587 scope.go:117] "RemoveContainer" containerID="611e58be21b580d3e3f769681fa2688e6fc51044b2db5dcfe3c9ba16ef583c66"
Sep 17 18:59:49.689722 ip-10-0-140-194 kubenswrapper[2587]: E0917 18:59:49.689702    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611e58be21b580d3e3f769681fa2688e6fc51044b2db5dcfe3c9ba16ef583c66\": container with ID starting with 611e58be21b580d3e3f769681fa2688e6fc51044b2db5dcfe3c9ba16ef583c66 not found: ID does not exist" containerID="611e58be21b580d3e3f769681fa2688e6fc51044b2db5dcfe3c9ba16ef583c66"
Sep 17 18:59:49.689770 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.689731    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611e58be21b580d3e3f769681fa2688e6fc51044b2db5dcfe3c9ba16ef583c66"} err="failed to get container status \"611e58be21b580d3e3f769681fa2688e6fc51044b2db5dcfe3c9ba16ef583c66\": rpc error: code = NotFound desc = could not find container \"611e58be21b580d3e3f769681fa2688e6fc51044b2db5dcfe3c9ba16ef583c66\": container with ID starting with 611e58be21b580d3e3f769681fa2688e6fc51044b2db5dcfe3c9ba16ef583c66 not found: ID does not exist"
Sep 17 18:59:49.726690 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.726655    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-dshm\") pod \"573d2e30-b1fb-45c9-87fe-623417f1c32b\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") "
Sep 17 18:59:49.726872 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.726707    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwdp4\" (UniqueName: \"kubernetes.io/projected/573d2e30-b1fb-45c9-87fe-623417f1c32b-kube-api-access-mwdp4\") pod \"573d2e30-b1fb-45c9-87fe-623417f1c32b\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") "
Sep 17 18:59:49.726872 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.726723    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-model-cache\") pod \"573d2e30-b1fb-45c9-87fe-623417f1c32b\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") "
Sep 17 18:59:49.726872 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.726745    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-home\") pod \"573d2e30-b1fb-45c9-87fe-623417f1c32b\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") "
Sep 17 18:59:49.727088 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.726882    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/573d2e30-b1fb-45c9-87fe-623417f1c32b-cabundle-cert\") pod \"573d2e30-b1fb-45c9-87fe-623417f1c32b\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") "
Sep 17 18:59:49.727088 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.726928    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-tmp-dir\") pod \"573d2e30-b1fb-45c9-87fe-623417f1c32b\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") "
Sep 17 18:59:49.727088 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.726999    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573d2e30-b1fb-45c9-87fe-623417f1c32b-tls-certs\") pod \"573d2e30-b1fb-45c9-87fe-623417f1c32b\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") "
Sep 17 18:59:49.727088 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.727024    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-kserve-provision-location\") pod \"573d2e30-b1fb-45c9-87fe-623417f1c32b\" (UID: \"573d2e30-b1fb-45c9-87fe-623417f1c32b\") "
Sep 17 18:59:49.727088 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.727036    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-model-cache" (OuterVolumeSpecName: "model-cache") pod "573d2e30-b1fb-45c9-87fe-623417f1c32b" (UID: "573d2e30-b1fb-45c9-87fe-623417f1c32b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:49.727341 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.727311    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:59:49.727400 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.727276    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573d2e30-b1fb-45c9-87fe-623417f1c32b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "573d2e30-b1fb-45c9-87fe-623417f1c32b" (UID: "573d2e30-b1fb-45c9-87fe-623417f1c32b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:59:49.727852 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.727825    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-home" (OuterVolumeSpecName: "home") pod "573d2e30-b1fb-45c9-87fe-623417f1c32b" (UID: "573d2e30-b1fb-45c9-87fe-623417f1c32b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:49.729308 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.729282    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-dshm" (OuterVolumeSpecName: "dshm") pod "573d2e30-b1fb-45c9-87fe-623417f1c32b" (UID: "573d2e30-b1fb-45c9-87fe-623417f1c32b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:49.729430 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.729411    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573d2e30-b1fb-45c9-87fe-623417f1c32b-kube-api-access-mwdp4" (OuterVolumeSpecName: "kube-api-access-mwdp4") pod "573d2e30-b1fb-45c9-87fe-623417f1c32b" (UID: "573d2e30-b1fb-45c9-87fe-623417f1c32b"). InnerVolumeSpecName "kube-api-access-mwdp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:59:49.729580 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.729557    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573d2e30-b1fb-45c9-87fe-623417f1c32b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "573d2e30-b1fb-45c9-87fe-623417f1c32b" (UID: "573d2e30-b1fb-45c9-87fe-623417f1c32b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:59:49.741730 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.741675    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "573d2e30-b1fb-45c9-87fe-623417f1c32b" (UID: "573d2e30-b1fb-45c9-87fe-623417f1c32b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:49.748195 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.748172    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "573d2e30-b1fb-45c9-87fe-623417f1c32b" (UID: "573d2e30-b1fb-45c9-87fe-623417f1c32b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:49.828503 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.828460    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/573d2e30-b1fb-45c9-87fe-623417f1c32b-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:59:49.828503 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.828495    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:59:49.828503 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.828507    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:59:49.828758 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.828520    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwdp4\" (UniqueName: \"kubernetes.io/projected/573d2e30-b1fb-45c9-87fe-623417f1c32b-kube-api-access-mwdp4\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:59:49.828758 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.828531    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:59:49.828758 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.828542    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/573d2e30-b1fb-45c9-87fe-623417f1c32b-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:59:49.828758 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.828552    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573d2e30-b1fb-45c9-87fe-623417f1c32b-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 18:59:49.995688 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:49.995647    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"]
Sep 17 18:59:50.001839 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:50.001810    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75dd8cfd66-vt89h"]
Sep 17 18:59:50.258128 ip-10-0-140-194 kubenswrapper[2587]: I0917 18:59:50.258094    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" path="/var/lib/kubelet/pods/573d2e30-b1fb-45c9-87fe-623417f1c32b/volumes"
Sep 17 19:00:11.415704 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:11.415664    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:12.796476 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:12.796426    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:18.392784 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:18.392743    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:23.636916 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:23.636877    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:23.738151 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:23.738107    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:24.623379 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:24.623345    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:24.723074 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:24.723047    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:28.805294 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:28.805254    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:29.699013 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:29.698983    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:35.351487 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:35.351448    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:35.450075 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:35.450044    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:38.626437 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.626396    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"]
Sep 17 19:00:38.628253 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.628227    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main"
Sep 17 19:00:38.628253 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.628254    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main"
Sep 17 19:00:38.628444 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.628289    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="storage-initializer"
Sep 17 19:00:38.628444 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.628312    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="storage-initializer"
Sep 17 19:00:38.628444 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.628335    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e07439eb-40ff-4205-b26d-9c909153f93a" containerName="main"
Sep 17 19:00:38.628444 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.628343    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="e07439eb-40ff-4205-b26d-9c909153f93a" containerName="main"
Sep 17 19:00:38.629205 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.629186    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="573d2e30-b1fb-45c9-87fe-623417f1c32b" containerName="main"
Sep 17 19:00:38.629288 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.629218    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="e07439eb-40ff-4205-b26d-9c909153f93a" containerName="main"
Sep 17 19:00:38.639880 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.639858    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.642275 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.642249    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:38.642737 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.642722    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 19:00:38.643080 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.643054    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 19:00:38.643197 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.643054    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 19:00:38.643933 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.643913    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-hrxhq\""
Sep 17 19:00:38.644147 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.643987    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:00:38.644228 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.644148    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 19:00:38.646979 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.646955    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"]
Sep 17 19:00:38.741929 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.741893    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.742136 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.741965    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-tls-certs\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.742136 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.741991    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.742136 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.742032    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.742136 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.742113    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2hqc\" (UniqueName: \"kubernetes.io/projected/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-kube-api-access-d2hqc\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.742282 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.742144    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-model-cache\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.742282 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.742163    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-dshm\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.742282 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.742180    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-home\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.843203 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.843143    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2hqc\" (UniqueName: \"kubernetes.io/projected/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-kube-api-access-d2hqc\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.843390 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.843226    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-model-cache\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.843390 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.843254    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-dshm\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.843390 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.843278    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-home\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.843390 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.843332    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.843390 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.843357    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-tls-certs\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.843390 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.843379    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.843796 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.843423    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.843796 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.843765    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-model-cache\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.843901 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.843793    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.843901 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.843836    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-home\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.843901 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.843874    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.844200 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.844184    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.845984 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.845962    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-dshm\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.846380 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.846358    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-tls-certs\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.858929 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.858901    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2hqc\" (UniqueName: \"kubernetes.io/projected/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-kube-api-access-d2hqc\") pod \"tokenizer-clean-path-test-kserve-86d755bd94-gtd7z\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:38.867569 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.867507    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"]
Sep 17 19:00:38.871963 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.871933    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:00:38.887406 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.887385    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"]
Sep 17 19:00:38.926097 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.926059    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:38.944147 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.944125    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:00:38.944276 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.944159    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:00:38.944276 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.944194    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:00:38.944276 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.944221    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:00:38.952361 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:38.952332    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:39.002643 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.002550    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:39.045302 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.045274    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:00:39.045479 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.045317    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:00:39.045479 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.045340    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:00:39.045479 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.045405    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:00:39.045675 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.045651    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:00:39.045738 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.045680    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:00:39.046072 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.046046    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:00:39.047921 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.047903    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:00:39.183081 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.182990    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:00:39.232918 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.232874    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"]
Sep 17 19:00:39.235421 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:00:39.235394    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c8c124_c237_4f7a_811f_f0ba52ec6d01.slice/crio-7ee54b2ed00bb20696965364436716371ebee927952496c918cce639d9aafa2d WatchSource:0}: Error finding container 7ee54b2ed00bb20696965364436716371ebee927952496c918cce639d9aafa2d: Status 404 returned error can't find the container with id 7ee54b2ed00bb20696965364436716371ebee927952496c918cce639d9aafa2d
Sep 17 19:00:39.456888 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.456782    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"]
Sep 17 19:00:39.460194 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:00:39.460159    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6ab920_53e3_4e4f_a21c_8e8881bbb18e.slice/crio-edc8d234e67a134fa72384f3a74d18a4845001e68ca8eb9d0e0fcebbd64533ed WatchSource:0}: Error finding container edc8d234e67a134fa72384f3a74d18a4845001e68ca8eb9d0e0fcebbd64533ed: Status 404 returned error can't find the container with id edc8d234e67a134fa72384f3a74d18a4845001e68ca8eb9d0e0fcebbd64533ed
Sep 17 19:00:39.896208 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.896170    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m" event={"ID":"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e","Type":"ContainerStarted","Data":"1918dc442c29fe951fe6a4da156461055b1306ce6c15b0e46a3c8d941bd7db06"}
Sep 17 19:00:39.896727 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.896234    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m" event={"ID":"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e","Type":"ContainerStarted","Data":"edc8d234e67a134fa72384f3a74d18a4845001e68ca8eb9d0e0fcebbd64533ed"}
Sep 17 19:00:39.897981 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.897956    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z" event={"ID":"b6c8c124-c237-4f7a-811f-f0ba52ec6d01","Type":"ContainerStarted","Data":"c3277fbb56273fa9eb9cf82f56522f172c709c85c676caa446097c0cd8f3ab37"}
Sep 17 19:00:39.898121 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:39.897998    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z" event={"ID":"b6c8c124-c237-4f7a-811f-f0ba52ec6d01","Type":"ContainerStarted","Data":"7ee54b2ed00bb20696965364436716371ebee927952496c918cce639d9aafa2d"}
Sep 17 19:00:41.910821 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:41.910789    2587 generic.go:356] "Generic (PLEG): container finished" podID="b6c8c124-c237-4f7a-811f-f0ba52ec6d01" containerID="c3277fbb56273fa9eb9cf82f56522f172c709c85c676caa446097c0cd8f3ab37" exitCode=0
Sep 17 19:00:41.911233 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:41.910839    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z" event={"ID":"b6c8c124-c237-4f7a-811f-f0ba52ec6d01","Type":"ContainerDied","Data":"c3277fbb56273fa9eb9cf82f56522f172c709c85c676caa446097c0cd8f3ab37"}
Sep 17 19:00:42.282240 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:42.282209    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:42.383205 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:42.383172    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:42.917566 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:42.917537    2587 generic.go:356] "Generic (PLEG): container finished" podID="7d6ab920-53e3-4e4f-a21c-8e8881bbb18e" containerID="1918dc442c29fe951fe6a4da156461055b1306ce6c15b0e46a3c8d941bd7db06" exitCode=0
Sep 17 19:00:42.918051 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:42.917615    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m" event={"ID":"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e","Type":"ContainerDied","Data":"1918dc442c29fe951fe6a4da156461055b1306ce6c15b0e46a3c8d941bd7db06"}
Sep 17 19:00:42.918823 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:42.918803    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:00:42.919691 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:42.919668    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z" event={"ID":"b6c8c124-c237-4f7a-811f-f0ba52ec6d01","Type":"ContainerStarted","Data":"775df419f4e7b2d55f2db3b696761ba9e4187b09a29a87c3606a12912cf3b854"}
Sep 17 19:00:42.953311 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:42.953280    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:42.965757 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:42.965713    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z" podStartSLOduration=2.414742297 podStartE2EDuration="4.965697745s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.550955448s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:00:42.963443857 +0000 UTC m=+1294.842412614" watchObservedRunningTime="2026-09-17 19:00:42.965697745 +0000 UTC m=+1294.844666502"
Sep 17 19:00:48.953042 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:48.953005    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:48.969224 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:48.969195    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:00:49.388892 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:49.388858    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:49.969013 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:00:49.968974    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:01:03.278178 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:01:03.278137    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:03.539436 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:01:03.539330    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:30.190087 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:01:30.190041    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m" event={"ID":"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e","Type":"ContainerStarted","Data":"1c20a4ee042edea1735f2eeafc93286497658fe4ac82d960d90b9a350c680d31"}
Sep 17 19:01:30.191889 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:01:30.191861    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:01:30.233476 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:01:30.233378    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m" podStartSLOduration=2.916384824 podStartE2EDuration="52.233350887s" totalImagesPullingTime="46.813626222s" totalInitContainerRuntime="2.503339841s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:01:30.233033845 +0000 UTC m=+1342.112002602" watchObservedRunningTime="2026-09-17 19:01:30.233350887 +0000 UTC m=+1342.112319642"
Sep 17 19:01:32.810959 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:01:32.810903    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:35.563465 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:01:35.563432    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:55.160477 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:01:55.160438    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:01.205874 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:01.205836    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:02:02.123841 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:02.123807    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m_7d6ab920-53e3-4e4f-a21c-8e8881bbb18e/main/0.log"
Sep 17 19:02:03.462783 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:03.462732    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:03.470394 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:03.470364    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"]
Sep 17 19:02:03.479757 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:02:03.479717    2587 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:02:03.479757 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:03.479757 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:03.479757 ip-10-0-140-194 kubenswrapper[2587]:  > containerID="775df419f4e7b2d55f2db3b696761ba9e4187b09a29a87c3606a12912cf3b854" cmd=["/bin/sleep","15"]
Sep 17 19:02:03.480041 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:03.479781    2587 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z" message="" err=<
Sep 17 19:02:03.480041 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:03.480041 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:03.480041 ip-10-0-140-194 kubenswrapper[2587]:  >
Sep 17 19:02:03.480041 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:02:03.479799    2587 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:02:03.480041 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:03.480041 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:03.480041 ip-10-0-140-194 kubenswrapper[2587]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z" podUID="b6c8c124-c237-4f7a-811f-f0ba52ec6d01" containerName="main" containerID="cri-o://775df419f4e7b2d55f2db3b696761ba9e4187b09a29a87c3606a12912cf3b854"
Sep 17 19:02:03.480041 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:03.479841    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z" podUID="b6c8c124-c237-4f7a-811f-f0ba52ec6d01" containerName="main" containerID="cri-o://775df419f4e7b2d55f2db3b696761ba9e4187b09a29a87c3606a12912cf3b854" gracePeriod=60
Sep 17 19:02:03.494977 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:03.494928    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"]
Sep 17 19:02:03.495192 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:03.495172    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m" podUID="7d6ab920-53e3-4e4f-a21c-8e8881bbb18e" containerName="main" containerID="cri-o://1c20a4ee042edea1735f2eeafc93286497658fe4ac82d960d90b9a350c680d31" gracePeriod=30
Sep 17 19:02:03.554098 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:03.554043    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:03.969179 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:03.969137    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z" podUID="b6c8c124-c237-4f7a-811f-f0ba52ec6d01" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Sep 17 19:02:04.185241 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.185201    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:02:04.235195 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.235160    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-kserve-provision-location\") pod \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") "
Sep 17 19:02:04.235397 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.235204    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-tls-certs\") pod \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") "
Sep 17 19:02:04.235397 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.235236    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-cabundle-cert\") pod \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") "
Sep 17 19:02:04.235397 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.235298    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-tmp-dir\") pod \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\" (UID: \"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e\") "
Sep 17 19:02:04.235823 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.235792    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d6ab920-53e3-4e4f-a21c-8e8881bbb18e" (UID: "7d6ab920-53e3-4e4f-a21c-8e8881bbb18e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:02:04.236125 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.235887    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d6ab920-53e3-4e4f-a21c-8e8881bbb18e" (UID: "7d6ab920-53e3-4e4f-a21c-8e8881bbb18e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:04.237705 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.237674    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d6ab920-53e3-4e4f-a21c-8e8881bbb18e" (UID: "7d6ab920-53e3-4e4f-a21c-8e8881bbb18e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:04.261833 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.261799    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d6ab920-53e3-4e4f-a21c-8e8881bbb18e" (UID: "7d6ab920-53e3-4e4f-a21c-8e8881bbb18e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:04.336306 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.336236    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.336306 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.336264    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.336306 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.336279    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.336306 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.336290    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.348158 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.348130    2587 generic.go:356] "Generic (PLEG): container finished" podID="7d6ab920-53e3-4e4f-a21c-8e8881bbb18e" containerID="1c20a4ee042edea1735f2eeafc93286497658fe4ac82d960d90b9a350c680d31" exitCode=0
Sep 17 19:02:04.348359 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.348182    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"
Sep 17 19:02:04.348359 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.348198    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m" event={"ID":"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e","Type":"ContainerDied","Data":"1c20a4ee042edea1735f2eeafc93286497658fe4ac82d960d90b9a350c680d31"}
Sep 17 19:02:04.348359 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.348246    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m" event={"ID":"7d6ab920-53e3-4e4f-a21c-8e8881bbb18e","Type":"ContainerDied","Data":"edc8d234e67a134fa72384f3a74d18a4845001e68ca8eb9d0e0fcebbd64533ed"}
Sep 17 19:02:04.348359 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.348275    2587 scope.go:117] "RemoveContainer" containerID="1c20a4ee042edea1735f2eeafc93286497658fe4ac82d960d90b9a350c680d31"
Sep 17 19:02:04.350056 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.350033    2587 generic.go:356] "Generic (PLEG): container finished" podID="b6c8c124-c237-4f7a-811f-f0ba52ec6d01" containerID="775df419f4e7b2d55f2db3b696761ba9e4187b09a29a87c3606a12912cf3b854" exitCode=0
Sep 17 19:02:04.350169 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.350081    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z" event={"ID":"b6c8c124-c237-4f7a-811f-f0ba52ec6d01","Type":"ContainerDied","Data":"775df419f4e7b2d55f2db3b696761ba9e4187b09a29a87c3606a12912cf3b854"}
Sep 17 19:02:04.350169 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.350121    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z" event={"ID":"b6c8c124-c237-4f7a-811f-f0ba52ec6d01","Type":"ContainerDied","Data":"7ee54b2ed00bb20696965364436716371ebee927952496c918cce639d9aafa2d"}
Sep 17 19:02:04.350169 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.350137    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ee54b2ed00bb20696965364436716371ebee927952496c918cce639d9aafa2d"
Sep 17 19:02:04.357839 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.357822    2587 scope.go:117] "RemoveContainer" containerID="1918dc442c29fe951fe6a4da156461055b1306ce6c15b0e46a3c8d941bd7db06"
Sep 17 19:02:04.366404 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.366382    2587 scope.go:117] "RemoveContainer" containerID="1c20a4ee042edea1735f2eeafc93286497658fe4ac82d960d90b9a350c680d31"
Sep 17 19:02:04.366713 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:02:04.366691    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c20a4ee042edea1735f2eeafc93286497658fe4ac82d960d90b9a350c680d31\": container with ID starting with 1c20a4ee042edea1735f2eeafc93286497658fe4ac82d960d90b9a350c680d31 not found: ID does not exist" containerID="1c20a4ee042edea1735f2eeafc93286497658fe4ac82d960d90b9a350c680d31"
Sep 17 19:02:04.366785 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.366726    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c20a4ee042edea1735f2eeafc93286497658fe4ac82d960d90b9a350c680d31"} err="failed to get container status \"1c20a4ee042edea1735f2eeafc93286497658fe4ac82d960d90b9a350c680d31\": rpc error: code = NotFound desc = could not find container \"1c20a4ee042edea1735f2eeafc93286497658fe4ac82d960d90b9a350c680d31\": container with ID starting with 1c20a4ee042edea1735f2eeafc93286497658fe4ac82d960d90b9a350c680d31 not found: ID does not exist"
Sep 17 19:02:04.366785 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.366753    2587 scope.go:117] "RemoveContainer" containerID="1918dc442c29fe951fe6a4da156461055b1306ce6c15b0e46a3c8d941bd7db06"
Sep 17 19:02:04.367037 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:02:04.367018    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1918dc442c29fe951fe6a4da156461055b1306ce6c15b0e46a3c8d941bd7db06\": container with ID starting with 1918dc442c29fe951fe6a4da156461055b1306ce6c15b0e46a3c8d941bd7db06 not found: ID does not exist" containerID="1918dc442c29fe951fe6a4da156461055b1306ce6c15b0e46a3c8d941bd7db06"
Sep 17 19:02:04.367091 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.367046    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1918dc442c29fe951fe6a4da156461055b1306ce6c15b0e46a3c8d941bd7db06"} err="failed to get container status \"1918dc442c29fe951fe6a4da156461055b1306ce6c15b0e46a3c8d941bd7db06\": rpc error: code = NotFound desc = could not find container \"1918dc442c29fe951fe6a4da156461055b1306ce6c15b0e46a3c8d941bd7db06\": container with ID starting with 1918dc442c29fe951fe6a4da156461055b1306ce6c15b0e46a3c8d941bd7db06 not found: ID does not exist"
Sep 17 19:02:04.372334 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.372315    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:02:04.386000 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.385968    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"]
Sep 17 19:02:04.390408 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.390383    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5846bb4449-m5q5m"]
Sep 17 19:02:04.437005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.436967    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2hqc\" (UniqueName: \"kubernetes.io/projected/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-kube-api-access-d2hqc\") pod \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") "
Sep 17 19:02:04.437005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.437011    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-dshm\") pod \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") "
Sep 17 19:02:04.437272 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.437047    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-cabundle-cert\") pod \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") "
Sep 17 19:02:04.437272 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.437136    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-tmp-dir\") pod \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") "
Sep 17 19:02:04.437272 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.437166    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-home\") pod \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") "
Sep 17 19:02:04.437272 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.437227    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-kserve-provision-location\") pod \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") "
Sep 17 19:02:04.437496 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.437281    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-model-cache\") pod \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") "
Sep 17 19:02:04.437496 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.437311    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-tls-certs\") pod \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\" (UID: \"b6c8c124-c237-4f7a-811f-f0ba52ec6d01\") "
Sep 17 19:02:04.437496 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.437437    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6c8c124-c237-4f7a-811f-f0ba52ec6d01" (UID: "b6c8c124-c237-4f7a-811f-f0ba52ec6d01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:04.437496 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.437489    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b6c8c124-c237-4f7a-811f-f0ba52ec6d01" (UID: "b6c8c124-c237-4f7a-811f-f0ba52ec6d01"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:02:04.437685 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.437553    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.437685 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.437572    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.437685 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.437625    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6c8c124-c237-4f7a-811f-f0ba52ec6d01" (UID: "b6c8c124-c237-4f7a-811f-f0ba52ec6d01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:04.438040 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.437688    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-home" (OuterVolumeSpecName: "home") pod "b6c8c124-c237-4f7a-811f-f0ba52ec6d01" (UID: "b6c8c124-c237-4f7a-811f-f0ba52ec6d01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:04.439490 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.439461    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-dshm" (OuterVolumeSpecName: "dshm") pod "b6c8c124-c237-4f7a-811f-f0ba52ec6d01" (UID: "b6c8c124-c237-4f7a-811f-f0ba52ec6d01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:04.439613 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.439594    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6c8c124-c237-4f7a-811f-f0ba52ec6d01" (UID: "b6c8c124-c237-4f7a-811f-f0ba52ec6d01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:04.440061 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.440034    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-kube-api-access-d2hqc" (OuterVolumeSpecName: "kube-api-access-d2hqc") pod "b6c8c124-c237-4f7a-811f-f0ba52ec6d01" (UID: "b6c8c124-c237-4f7a-811f-f0ba52ec6d01"). InnerVolumeSpecName "kube-api-access-d2hqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:02:04.458584 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.458521    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6c8c124-c237-4f7a-811f-f0ba52ec6d01" (UID: "b6c8c124-c237-4f7a-811f-f0ba52ec6d01"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:04.538018 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.537984    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.538018 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.538018    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.538018 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.538028    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2hqc\" (UniqueName: \"kubernetes.io/projected/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-kube-api-access-d2hqc\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.538490 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.538039    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.538490 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.538049    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:02:04.538490 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:04.538056    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6c8c124-c237-4f7a-811f-f0ba52ec6d01-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:02:05.356473 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:05.356441    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"
Sep 17 19:02:05.379132 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:05.379097    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"]
Sep 17 19:02:05.384458 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:05.384429    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-86d755bd94-gtd7z"]
Sep 17 19:02:06.257817 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:06.257769    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d6ab920-53e3-4e4f-a21c-8e8881bbb18e" path="/var/lib/kubelet/pods/7d6ab920-53e3-4e4f-a21c-8e8881bbb18e/volumes"
Sep 17 19:02:06.258437 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:06.258416    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6c8c124-c237-4f7a-811f-f0ba52ec6d01" path="/var/lib/kubelet/pods/b6c8c124-c237-4f7a-811f-f0ba52ec6d01/volumes"
Sep 17 19:02:52.459388 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:52.459291    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:52.591168 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:52.591108    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:59.180831 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:02:59.180785    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:04.585760 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.585720    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"]
Sep 17 19:03:04.586170 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.586047    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c8c124-c237-4f7a-811f-f0ba52ec6d01" containerName="storage-initializer"
Sep 17 19:03:04.586170 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.586059    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c8c124-c237-4f7a-811f-f0ba52ec6d01" containerName="storage-initializer"
Sep 17 19:03:04.586170 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.586067    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d6ab920-53e3-4e4f-a21c-8e8881bbb18e" containerName="storage-initializer"
Sep 17 19:03:04.586170 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.586072    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d6ab920-53e3-4e4f-a21c-8e8881bbb18e" containerName="storage-initializer"
Sep 17 19:03:04.586170 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.586079    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d6ab920-53e3-4e4f-a21c-8e8881bbb18e" containerName="main"
Sep 17 19:03:04.586170 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.586085    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d6ab920-53e3-4e4f-a21c-8e8881bbb18e" containerName="main"
Sep 17 19:03:04.586170 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.586093    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c8c124-c237-4f7a-811f-f0ba52ec6d01" containerName="main"
Sep 17 19:03:04.586170 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.586098    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c8c124-c237-4f7a-811f-f0ba52ec6d01" containerName="main"
Sep 17 19:03:04.586170 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.586172    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d6ab920-53e3-4e4f-a21c-8e8881bbb18e" containerName="main"
Sep 17 19:03:04.586475 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.586181    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6c8c124-c237-4f7a-811f-f0ba52ec6d01" containerName="main"
Sep 17 19:03:04.588163 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.588144    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.590835 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.590811    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 19:03:04.590991 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.590901    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 19:03:04.591257 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.591229    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:03:04.591547 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.591533    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-4nf2q\""
Sep 17 19:03:04.592246 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.592216    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 19:03:04.592353 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.592271    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 19:03:04.602565 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.602539    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"]
Sep 17 19:03:04.631434 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.631405    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:04.738806 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.738770    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-tls-certs\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.738806 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.738814    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.739133 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.738884    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-dshm\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.739133 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.738921    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-model-cache\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.739133 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.739010    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.739133 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.739040    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-646kf\" (UniqueName: \"kubernetes.io/projected/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-kube-api-access-646kf\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.739133 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.739067    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.739133 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.739108    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-home\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.840524 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.840418    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-dshm\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.840524 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.840465    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-model-cache\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.840524 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.840497    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.840524 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.840522    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-646kf\" (UniqueName: \"kubernetes.io/projected/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-kube-api-access-646kf\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.840861 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.840547    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.840861 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.840569    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-home\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.840861 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.840718    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-tls-certs\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.840861 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.840770    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.841105 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.840969    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-model-cache\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.841105 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.841081    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.841210 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.841162    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.841378 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.841358    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-home\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.841559 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.841537    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.843059 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.843040    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-dshm\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.843398 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.843379    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-tls-certs\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.856452 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.856416    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-646kf\" (UniqueName: \"kubernetes.io/projected/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-kube-api-access-646kf\") pod \"tokenizer-migration-path-test-kserve-b4f5d5767-j79b8\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.866607 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.866579    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"]
Sep 17 19:03:04.869507 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.869485    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:04.881914 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.881889    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"]
Sep 17 19:03:04.899904 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.899870    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:04.931695 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:04.931661    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:05.006552 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.006520    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:05.042788 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.042757    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/788aa05c-fa00-411f-a13c-374d3fdc8214-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:05.042927 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.042835    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/788aa05c-fa00-411f-a13c-374d3fdc8214-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:05.042927 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.042859    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/788aa05c-fa00-411f-a13c-374d3fdc8214-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:05.042927 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.042883    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/788aa05c-fa00-411f-a13c-374d3fdc8214-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:05.143650 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.143549    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/788aa05c-fa00-411f-a13c-374d3fdc8214-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:05.143650 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.143633    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/788aa05c-fa00-411f-a13c-374d3fdc8214-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:05.143650 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.143652    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/788aa05c-fa00-411f-a13c-374d3fdc8214-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:05.143982 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.143677    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/788aa05c-fa00-411f-a13c-374d3fdc8214-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:05.144053 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.143994    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/788aa05c-fa00-411f-a13c-374d3fdc8214-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:05.144053 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.144025    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/788aa05c-fa00-411f-a13c-374d3fdc8214-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:05.144343 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.144319    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/788aa05c-fa00-411f-a13c-374d3fdc8214-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:05.146403 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.146380    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/788aa05c-fa00-411f-a13c-374d3fdc8214-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:05.174834 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.174803    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:05.180144 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.180122    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:05.190102 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.190049    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"]
Sep 17 19:03:05.192234 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:03:05.192210    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9bffb8_9688_4fd6_8cc3_031055aa79b0.slice/crio-2365b5a9236097e21e4e0fa03e6753189f88be2400215ce97e919e763bae99b0 WatchSource:0}: Error finding container 2365b5a9236097e21e4e0fa03e6753189f88be2400215ce97e919e763bae99b0: Status 404 returned error can't find the container with id 2365b5a9236097e21e4e0fa03e6753189f88be2400215ce97e919e763bae99b0
Sep 17 19:03:05.276914 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.276885    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:05.463314 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.463280    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"]
Sep 17 19:03:05.464771 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:03:05.464740    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788aa05c_fa00_411f_a13c_374d3fdc8214.slice/crio-39e402df3875d361f1931c10cbb5b4153a643baab76a96d2a0b5e49404ed1d9c WatchSource:0}: Error finding container 39e402df3875d361f1931c10cbb5b4153a643baab76a96d2a0b5e49404ed1d9c: Status 404 returned error can't find the container with id 39e402df3875d361f1931c10cbb5b4153a643baab76a96d2a0b5e49404ed1d9c
Sep 17 19:03:05.628749 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.628650    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7" event={"ID":"788aa05c-fa00-411f-a13c-374d3fdc8214","Type":"ContainerStarted","Data":"a69f31873f6cde1ed04f0cddeac5de02f71fdd7c701746176cb3afd4b7bb0c6d"}
Sep 17 19:03:05.628749 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.628728    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7" event={"ID":"788aa05c-fa00-411f-a13c-374d3fdc8214","Type":"ContainerStarted","Data":"39e402df3875d361f1931c10cbb5b4153a643baab76a96d2a0b5e49404ed1d9c"}
Sep 17 19:03:05.630392 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.630364    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8" event={"ID":"3b9bffb8-9688-4fd6-8cc3-031055aa79b0","Type":"ContainerStarted","Data":"66a4e83e9435cf50e313056c88d075a4aef1a671e994b8425e46aea0496c66c1"}
Sep 17 19:03:05.630544 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:05.630408    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8" event={"ID":"3b9bffb8-9688-4fd6-8cc3-031055aa79b0","Type":"ContainerStarted","Data":"2365b5a9236097e21e4e0fa03e6753189f88be2400215ce97e919e763bae99b0"}
Sep 17 19:03:08.645571 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:08.645540    2587 generic.go:356] "Generic (PLEG): container finished" podID="788aa05c-fa00-411f-a13c-374d3fdc8214" containerID="a69f31873f6cde1ed04f0cddeac5de02f71fdd7c701746176cb3afd4b7bb0c6d" exitCode=0
Sep 17 19:03:08.646038 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:08.645625    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7" event={"ID":"788aa05c-fa00-411f-a13c-374d3fdc8214","Type":"ContainerDied","Data":"a69f31873f6cde1ed04f0cddeac5de02f71fdd7c701746176cb3afd4b7bb0c6d"}
Sep 17 19:03:08.647305 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:08.647284    2587 generic.go:356] "Generic (PLEG): container finished" podID="3b9bffb8-9688-4fd6-8cc3-031055aa79b0" containerID="66a4e83e9435cf50e313056c88d075a4aef1a671e994b8425e46aea0496c66c1" exitCode=0
Sep 17 19:03:08.647381 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:08.647357    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8" event={"ID":"3b9bffb8-9688-4fd6-8cc3-031055aa79b0","Type":"ContainerDied","Data":"66a4e83e9435cf50e313056c88d075a4aef1a671e994b8425e46aea0496c66c1"}
Sep 17 19:03:09.654181 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:09.654147    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7" event={"ID":"788aa05c-fa00-411f-a13c-374d3fdc8214","Type":"ContainerStarted","Data":"04f4c2af5734839ff0b0687031601e616b745599d31023ccd23bf2252701e2b0"}
Sep 17 19:03:09.654658 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:09.654386    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:09.656284 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:09.656260    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8" event={"ID":"3b9bffb8-9688-4fd6-8cc3-031055aa79b0","Type":"ContainerStarted","Data":"360e62574d2b0ff0be01f40e29b569165e73615dc122446c3dd314ef7ac9c532"}
Sep 17 19:03:09.677384 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:09.677326    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7" podStartSLOduration=3.113367274 podStartE2EDuration="5.677307934s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.56394066s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:03:09.675635848 +0000 UTC m=+1441.554604606" watchObservedRunningTime="2026-09-17 19:03:09.677307934 +0000 UTC m=+1441.556276687"
Sep 17 19:03:09.699466 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:09.699401    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8" podStartSLOduration=3.1364065 podStartE2EDuration="5.699381712s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.562975212s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:03:09.697543696 +0000 UTC m=+1441.576512453" watchObservedRunningTime="2026-09-17 19:03:09.699381712 +0000 UTC m=+1441.578350468"
Sep 17 19:03:09.900057 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:09.900017    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:13.986509 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:13.986465    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:14.900112 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:14.900049    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:14.913656 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:14.913622    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:15.698865 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:15.698837    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:40.673389 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:40.673354    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:42.033100 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:42.033066    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7_788aa05c-fa00-411f-a13c-374d3fdc8214/main/0.log"
Sep 17 19:03:42.247865 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:42.247832    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:42.260891 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:42.260857    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"]
Sep 17 19:03:42.261128 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:42.261103    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7" podUID="788aa05c-fa00-411f-a13c-374d3fdc8214" containerName="main" containerID="cri-o://04f4c2af5734839ff0b0687031601e616b745599d31023ccd23bf2252701e2b0" gracePeriod=30
Sep 17 19:03:42.274044 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:42.274012    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"]
Sep 17 19:03:42.283666 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:03:42.283574    2587 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:03:42.283666 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:03:42.283666 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:03:42.283666 ip-10-0-140-194 kubenswrapper[2587]:  > containerID="360e62574d2b0ff0be01f40e29b569165e73615dc122446c3dd314ef7ac9c532" cmd=["/bin/sleep","15"]
Sep 17 19:03:42.283666 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:42.283638    2587 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8" message="" err=<
Sep 17 19:03:42.283666 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:03:42.283666 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:03:42.283666 ip-10-0-140-194 kubenswrapper[2587]:  >
Sep 17 19:03:42.283666 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:03:42.283655    2587 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:03:42.283666 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:03:42.283666 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:03:42.283666 ip-10-0-140-194 kubenswrapper[2587]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8" podUID="3b9bffb8-9688-4fd6-8cc3-031055aa79b0" containerName="main" containerID="cri-o://360e62574d2b0ff0be01f40e29b569165e73615dc122446c3dd314ef7ac9c532"
Sep 17 19:03:42.284469 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:42.283703    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8" podUID="3b9bffb8-9688-4fd6-8cc3-031055aa79b0" containerName="main" containerID="cri-o://360e62574d2b0ff0be01f40e29b569165e73615dc122446c3dd314ef7ac9c532" gracePeriod=60
Sep 17 19:03:42.331626 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:42.331588    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:42.687985 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:42.687856    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8" podUID="3b9bffb8-9688-4fd6-8cc3-031055aa79b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 17 19:03:42.812676 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:42.812646    2587 generic.go:356] "Generic (PLEG): container finished" podID="788aa05c-fa00-411f-a13c-374d3fdc8214" containerID="04f4c2af5734839ff0b0687031601e616b745599d31023ccd23bf2252701e2b0" exitCode=0
Sep 17 19:03:42.812862 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:42.812721    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7" event={"ID":"788aa05c-fa00-411f-a13c-374d3fdc8214","Type":"ContainerDied","Data":"04f4c2af5734839ff0b0687031601e616b745599d31023ccd23bf2252701e2b0"}
Sep 17 19:03:42.814873 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:42.814848    2587 generic.go:356] "Generic (PLEG): container finished" podID="3b9bffb8-9688-4fd6-8cc3-031055aa79b0" containerID="360e62574d2b0ff0be01f40e29b569165e73615dc122446c3dd314ef7ac9c532" exitCode=0
Sep 17 19:03:42.815020 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:42.814915    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8" event={"ID":"3b9bffb8-9688-4fd6-8cc3-031055aa79b0","Type":"ContainerDied","Data":"360e62574d2b0ff0be01f40e29b569165e73615dc122446c3dd314ef7ac9c532"}
Sep 17 19:03:42.915145 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:42.915109    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:43.037686 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.037661    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:43.077591 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.077559    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/788aa05c-fa00-411f-a13c-374d3fdc8214-kserve-provision-location\") pod \"788aa05c-fa00-411f-a13c-374d3fdc8214\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") "
Sep 17 19:03:43.077779 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.077608    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/788aa05c-fa00-411f-a13c-374d3fdc8214-tmp-dir\") pod \"788aa05c-fa00-411f-a13c-374d3fdc8214\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") "
Sep 17 19:03:43.077779 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.077653    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/788aa05c-fa00-411f-a13c-374d3fdc8214-tls-certs\") pod \"788aa05c-fa00-411f-a13c-374d3fdc8214\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") "
Sep 17 19:03:43.077779 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.077673    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/788aa05c-fa00-411f-a13c-374d3fdc8214-cabundle-cert\") pod \"788aa05c-fa00-411f-a13c-374d3fdc8214\" (UID: \"788aa05c-fa00-411f-a13c-374d3fdc8214\") "
Sep 17 19:03:43.078127 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.078097    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788aa05c-fa00-411f-a13c-374d3fdc8214-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "788aa05c-fa00-411f-a13c-374d3fdc8214" (UID: "788aa05c-fa00-411f-a13c-374d3fdc8214"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:43.078203 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.078115    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788aa05c-fa00-411f-a13c-374d3fdc8214-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "788aa05c-fa00-411f-a13c-374d3fdc8214" (UID: "788aa05c-fa00-411f-a13c-374d3fdc8214"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:03:43.080280 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.080250    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/788aa05c-fa00-411f-a13c-374d3fdc8214-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "788aa05c-fa00-411f-a13c-374d3fdc8214" (UID: "788aa05c-fa00-411f-a13c-374d3fdc8214"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:03:43.099144 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.099089    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788aa05c-fa00-411f-a13c-374d3fdc8214-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "788aa05c-fa00-411f-a13c-374d3fdc8214" (UID: "788aa05c-fa00-411f-a13c-374d3fdc8214"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:43.179157 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179115    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-kserve-provision-location\") pod \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") "
Sep 17 19:03:43.179331 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179165    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-tls-certs\") pod \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") "
Sep 17 19:03:43.179331 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179189    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-tmp-dir\") pod \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") "
Sep 17 19:03:43.179331 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179222    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-cabundle-cert\") pod \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") "
Sep 17 19:03:43.179331 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179251    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-home\") pod \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") "
Sep 17 19:03:43.179331 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179284    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-646kf\" (UniqueName: \"kubernetes.io/projected/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-kube-api-access-646kf\") pod \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") "
Sep 17 19:03:43.179331 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179333    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-dshm\") pod \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") "
Sep 17 19:03:43.179643 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179364    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-model-cache\") pod \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\" (UID: \"3b9bffb8-9688-4fd6-8cc3-031055aa79b0\") "
Sep 17 19:03:43.179643 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179494    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b9bffb8-9688-4fd6-8cc3-031055aa79b0" (UID: "3b9bffb8-9688-4fd6-8cc3-031055aa79b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:43.179643 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179572    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-home" (OuterVolumeSpecName: "home") pod "3b9bffb8-9688-4fd6-8cc3-031055aa79b0" (UID: "3b9bffb8-9688-4fd6-8cc3-031055aa79b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:43.179643 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179620    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b9bffb8-9688-4fd6-8cc3-031055aa79b0" (UID: "3b9bffb8-9688-4fd6-8cc3-031055aa79b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:03:43.179824 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179660    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.179824 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179678    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.179824 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179692    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/788aa05c-fa00-411f-a13c-374d3fdc8214-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.179824 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179705    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/788aa05c-fa00-411f-a13c-374d3fdc8214-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.179824 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179718    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/788aa05c-fa00-411f-a13c-374d3fdc8214-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.179824 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179730    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/788aa05c-fa00-411f-a13c-374d3fdc8214-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.179824 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.179757    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b9bffb8-9688-4fd6-8cc3-031055aa79b0" (UID: "3b9bffb8-9688-4fd6-8cc3-031055aa79b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:43.181605 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.181581    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b9bffb8-9688-4fd6-8cc3-031055aa79b0" (UID: "3b9bffb8-9688-4fd6-8cc3-031055aa79b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:03:43.181747 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.181725    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-dshm" (OuterVolumeSpecName: "dshm") pod "3b9bffb8-9688-4fd6-8cc3-031055aa79b0" (UID: "3b9bffb8-9688-4fd6-8cc3-031055aa79b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:43.181791 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.181741    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-kube-api-access-646kf" (OuterVolumeSpecName: "kube-api-access-646kf") pod "3b9bffb8-9688-4fd6-8cc3-031055aa79b0" (UID: "3b9bffb8-9688-4fd6-8cc3-031055aa79b0"). InnerVolumeSpecName "kube-api-access-646kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:03:43.201588 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.201539    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b9bffb8-9688-4fd6-8cc3-031055aa79b0" (UID: "3b9bffb8-9688-4fd6-8cc3-031055aa79b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:43.280779 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.280745    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.280779 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.280774    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.280779 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.280785    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.281036 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.280794    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.281036 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.280804    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.281036 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.280812    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-646kf\" (UniqueName: \"kubernetes.io/projected/3b9bffb8-9688-4fd6-8cc3-031055aa79b0-kube-api-access-646kf\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.821177 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.821135    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7" event={"ID":"788aa05c-fa00-411f-a13c-374d3fdc8214","Type":"ContainerDied","Data":"39e402df3875d361f1931c10cbb5b4153a643baab76a96d2a0b5e49404ed1d9c"}
Sep 17 19:03:43.821362 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.821159    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"
Sep 17 19:03:43.821362 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.821212    2587 scope.go:117] "RemoveContainer" containerID="04f4c2af5734839ff0b0687031601e616b745599d31023ccd23bf2252701e2b0"
Sep 17 19:03:43.823161 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.823124    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8" event={"ID":"3b9bffb8-9688-4fd6-8cc3-031055aa79b0","Type":"ContainerDied","Data":"2365b5a9236097e21e4e0fa03e6753189f88be2400215ce97e919e763bae99b0"}
Sep 17 19:03:43.823261 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.823172    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"
Sep 17 19:03:43.830414 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.830395    2587 scope.go:117] "RemoveContainer" containerID="a69f31873f6cde1ed04f0cddeac5de02f71fdd7c701746176cb3afd4b7bb0c6d"
Sep 17 19:03:43.842834 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.842811    2587 scope.go:117] "RemoveContainer" containerID="360e62574d2b0ff0be01f40e29b569165e73615dc122446c3dd314ef7ac9c532"
Sep 17 19:03:43.851256 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.851234    2587 scope.go:117] "RemoveContainer" containerID="66a4e83e9435cf50e313056c88d075a4aef1a671e994b8425e46aea0496c66c1"
Sep 17 19:03:43.857784 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.857759    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"]
Sep 17 19:03:43.862262 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.862240    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-b4f5d5767-j79b8"]
Sep 17 19:03:43.873245 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.873218    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"]
Sep 17 19:03:43.878134 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:43.878113    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5c77f6dc-v5cw7"]
Sep 17 19:03:44.258486 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:44.258450    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b9bffb8-9688-4fd6-8cc3-031055aa79b0" path="/var/lib/kubelet/pods/3b9bffb8-9688-4fd6-8cc3-031055aa79b0/volumes"
Sep 17 19:03:44.258874 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:44.258866    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="788aa05c-fa00-411f-a13c-374d3fdc8214" path="/var/lib/kubelet/pods/788aa05c-fa00-411f-a13c-374d3fdc8214/volumes"
Sep 17 19:03:46.272233 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:46.272203    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:46.367737 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:03:46.367703    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:10.251208 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:10.251177    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:04:10.252770 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:10.252738    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:04:33.104773 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:33.104668    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:33.204848 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:33.204813    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:35.815865 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:35.815832    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:35.949270 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:35.949227    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:39.275002 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:39.274934    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:42.197113 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:42.197081    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:44.881132 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:44.881096    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:44.962958 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:44.962907    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:47.865691 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:47.865656    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:48.069829 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:48.069767    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:57.578918 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.578880    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"]
Sep 17 19:04:57.580349 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.579358    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="788aa05c-fa00-411f-a13c-374d3fdc8214" containerName="main"
Sep 17 19:04:57.580349 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.579373    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="788aa05c-fa00-411f-a13c-374d3fdc8214" containerName="main"
Sep 17 19:04:57.580349 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.579390    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b9bffb8-9688-4fd6-8cc3-031055aa79b0" containerName="storage-initializer"
Sep 17 19:04:57.580349 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.579396    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b9bffb8-9688-4fd6-8cc3-031055aa79b0" containerName="storage-initializer"
Sep 17 19:04:57.580349 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.579402    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="788aa05c-fa00-411f-a13c-374d3fdc8214" containerName="storage-initializer"
Sep 17 19:04:57.580349 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.579407    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="788aa05c-fa00-411f-a13c-374d3fdc8214" containerName="storage-initializer"
Sep 17 19:04:57.580349 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.579416    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b9bffb8-9688-4fd6-8cc3-031055aa79b0" containerName="main"
Sep 17 19:04:57.580349 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.579422    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b9bffb8-9688-4fd6-8cc3-031055aa79b0" containerName="main"
Sep 17 19:04:57.580349 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.579475    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b9bffb8-9688-4fd6-8cc3-031055aa79b0" containerName="main"
Sep 17 19:04:57.580349 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.579483    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="788aa05c-fa00-411f-a13c-374d3fdc8214" containerName="main"
Sep 17 19:04:57.581376 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.581357    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.584172 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.584152    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 19:04:57.585114 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.585091    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-gzg4z\""
Sep 17 19:04:57.585224 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.585152    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 19:04:57.585224 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.585208    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 19:04:57.585224 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.585217    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:04:57.585460 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.585444    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 19:04:57.597446 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.597421    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"]
Sep 17 19:04:57.610527 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.610500    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:57.622990 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.622962    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-model-cache\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.623113 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.622996    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-tmp-dir\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.623113 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.623021    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-home\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.623113 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.623095    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c23bc66-b192-4813-aa74-ea6166e422d8-tls-certs\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.623230 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.623148    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c23bc66-b192-4813-aa74-ea6166e422d8-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.623230 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.623186    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwqfb\" (UniqueName: \"kubernetes.io/projected/8c23bc66-b192-4813-aa74-ea6166e422d8-kube-api-access-zwqfb\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.623230 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.623218    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-dshm\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.623325 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.623262    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.724368 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.724333    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c23bc66-b192-4813-aa74-ea6166e422d8-tls-certs\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.724561 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.724380    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c23bc66-b192-4813-aa74-ea6166e422d8-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.724561 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.724400    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqfb\" (UniqueName: \"kubernetes.io/projected/8c23bc66-b192-4813-aa74-ea6166e422d8-kube-api-access-zwqfb\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.724561 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.724418    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-dshm\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.724739 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.724560    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.724739 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.724629    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-model-cache\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.724739 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.724664    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-tmp-dir\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.724739 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.724701    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-home\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.725170 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.725079    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.725170 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.725120    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-home\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.725170 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.725154    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-model-cache\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.725325 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.725181    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-tmp-dir\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.725325 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.725268    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c23bc66-b192-4813-aa74-ea6166e422d8-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.727525 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.727504    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-dshm\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.727712 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.727695    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c23bc66-b192-4813-aa74-ea6166e422d8-tls-certs\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.735607 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.735581    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqfb\" (UniqueName: \"kubernetes.io/projected/8c23bc66-b192-4813-aa74-ea6166e422d8-kube-api-access-zwqfb\") pod \"tokenizer-template-args-test-kserve-87bfb5b97-46kqm\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.814434 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.814400    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"]
Sep 17 19:04:57.817553 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.817532    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:04:57.830249 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.830174    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"]
Sep 17 19:04:57.870509 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.870478    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:57.891106 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.891067    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:04:57.927074 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.927027    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2df6b1a7-573f-409d-b535-a8af086eba03-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-64876b6f79-shsgf\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:04:57.927247 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.927106    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df6b1a7-573f-409d-b535-a8af086eba03-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-64876b6f79-shsgf\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:04:57.927247 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.927168    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df6b1a7-573f-409d-b535-a8af086eba03-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-64876b6f79-shsgf\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:04:57.927247 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.927227    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df6b1a7-573f-409d-b535-a8af086eba03-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-64876b6f79-shsgf\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:04:57.941839 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:57.941561    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:58.028622 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:58.028578    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df6b1a7-573f-409d-b535-a8af086eba03-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-64876b6f79-shsgf\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:04:58.028809 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:58.028644    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2df6b1a7-573f-409d-b535-a8af086eba03-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-64876b6f79-shsgf\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:04:58.028809 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:58.028697    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df6b1a7-573f-409d-b535-a8af086eba03-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-64876b6f79-shsgf\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:04:58.028809 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:58.028731    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df6b1a7-573f-409d-b535-a8af086eba03-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-64876b6f79-shsgf\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:04:58.029154 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:58.029132    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df6b1a7-573f-409d-b535-a8af086eba03-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-64876b6f79-shsgf\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:04:58.029214 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:58.029173    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df6b1a7-573f-409d-b535-a8af086eba03-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-64876b6f79-shsgf\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:04:58.029471 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:58.029446    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2df6b1a7-573f-409d-b535-a8af086eba03-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-64876b6f79-shsgf\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:04:58.031398 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:58.031381    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df6b1a7-573f-409d-b535-a8af086eba03-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-64876b6f79-shsgf\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:04:58.129805 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:58.129714    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:04:58.166688 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:58.166657    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"]
Sep 17 19:04:58.170825 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:04:58.170799    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c23bc66_b192_4813_aa74_ea6166e422d8.slice/crio-0e4f534d04bdfe8b1690a2def2ec8ca1859d3e4a078ab93a796544aac121b3d1 WatchSource:0}: Error finding container 0e4f534d04bdfe8b1690a2def2ec8ca1859d3e4a078ab93a796544aac121b3d1: Status 404 returned error can't find the container with id 0e4f534d04bdfe8b1690a2def2ec8ca1859d3e4a078ab93a796544aac121b3d1
Sep 17 19:04:58.404920 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:58.404890    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"]
Sep 17 19:04:58.405899 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:04:58.405877    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df6b1a7_573f_409d_b535_a8af086eba03.slice/crio-c4c6fabe7ac363620738f585fa2c010253850553dc30bbecd5f33b9c87acb8cd WatchSource:0}: Error finding container c4c6fabe7ac363620738f585fa2c010253850553dc30bbecd5f33b9c87acb8cd: Status 404 returned error can't find the container with id c4c6fabe7ac363620738f585fa2c010253850553dc30bbecd5f33b9c87acb8cd
Sep 17 19:04:59.167931 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:59.167895    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf" event={"ID":"2df6b1a7-573f-409d-b535-a8af086eba03","Type":"ContainerStarted","Data":"15908e9f94c0f106228bdfcea6a2b3ddb9c4b74d7744bd4e53368940a9c8bb5c"}
Sep 17 19:04:59.168447 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:59.168008    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf" event={"ID":"2df6b1a7-573f-409d-b535-a8af086eba03","Type":"ContainerStarted","Data":"c4c6fabe7ac363620738f585fa2c010253850553dc30bbecd5f33b9c87acb8cd"}
Sep 17 19:04:59.169652 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:59.169626    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm" event={"ID":"8c23bc66-b192-4813-aa74-ea6166e422d8","Type":"ContainerStarted","Data":"042acf105c62b1090fe150a7ea613b74d5089ffde79eddd3e60bdc6012645946"}
Sep 17 19:04:59.169759 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:04:59.169675    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm" event={"ID":"8c23bc66-b192-4813-aa74-ea6166e422d8","Type":"ContainerStarted","Data":"0e4f534d04bdfe8b1690a2def2ec8ca1859d3e4a078ab93a796544aac121b3d1"}
Sep 17 19:05:01.180062 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:01.179971    2587 generic.go:356] "Generic (PLEG): container finished" podID="2df6b1a7-573f-409d-b535-a8af086eba03" containerID="15908e9f94c0f106228bdfcea6a2b3ddb9c4b74d7744bd4e53368940a9c8bb5c" exitCode=0
Sep 17 19:05:01.180062 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:01.180042    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf" event={"ID":"2df6b1a7-573f-409d-b535-a8af086eba03","Type":"ContainerDied","Data":"15908e9f94c0f106228bdfcea6a2b3ddb9c4b74d7744bd4e53368940a9c8bb5c"}
Sep 17 19:05:01.181727 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:01.181703    2587 generic.go:356] "Generic (PLEG): container finished" podID="8c23bc66-b192-4813-aa74-ea6166e422d8" containerID="042acf105c62b1090fe150a7ea613b74d5089ffde79eddd3e60bdc6012645946" exitCode=0
Sep 17 19:05:01.181833 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:01.181755    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm" event={"ID":"8c23bc66-b192-4813-aa74-ea6166e422d8","Type":"ContainerDied","Data":"042acf105c62b1090fe150a7ea613b74d5089ffde79eddd3e60bdc6012645946"}
Sep 17 19:05:02.189661 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:02.189625    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf" event={"ID":"2df6b1a7-573f-409d-b535-a8af086eba03","Type":"ContainerStarted","Data":"71b7651902ae14ecc0314d94fbe6ce64cefbffb2870585c15b1990851db199a2"}
Sep 17 19:05:02.190109 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:02.189889    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:05:02.191767 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:02.191747    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm" event={"ID":"8c23bc66-b192-4813-aa74-ea6166e422d8","Type":"ContainerStarted","Data":"caf9a50a942116fb9a4b29c6d36c329db60f0de346232d7fb74a7f70f8fb9659"}
Sep 17 19:05:02.211238 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:02.211190    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf" podStartSLOduration=2.629670634 podStartE2EDuration="5.211174931s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.581504297s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:05:02.209889051 +0000 UTC m=+1554.088857809" watchObservedRunningTime="2026-09-17 19:05:02.211174931 +0000 UTC m=+1554.090143687"
Sep 17 19:05:02.234823 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:02.234769    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm" podStartSLOduration=2.654403791 podStartE2EDuration="5.234751234s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.580347443s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:05:02.23183881 +0000 UTC m=+1554.110807566" watchObservedRunningTime="2026-09-17 19:05:02.234751234 +0000 UTC m=+1554.113719991"
Sep 17 19:05:02.891690 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:02.891656    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:05:06.437024 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:06.436917    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:07.891269 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:07.891225    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:05:07.904626 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:07.904601    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:05:08.233499 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:08.233413    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:05:18.053693 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:18.053657    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:19.552259 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:19.552221    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:19.651926 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:19.651892    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:33.209344 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:33.209309    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:05:46.529211 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:46.529177    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:49.763878 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:49.763777    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-64876b6f79-shsgf_2df6b1a7-573f-409d-b535-a8af086eba03/main/0.log"
Sep 17 19:05:52.275166 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.275119    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:52.293181 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.293150    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"]
Sep 17 19:05:52.303170 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:05:52.303109    2587 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:05:52.303170 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:05:52.303170 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:05:52.303170 ip-10-0-140-194 kubenswrapper[2587]:  > containerID="caf9a50a942116fb9a4b29c6d36c329db60f0de346232d7fb74a7f70f8fb9659" cmd=["/bin/sleep","15"]
Sep 17 19:05:52.303426 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.303204    2587 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm" message="" err=<
Sep 17 19:05:52.303426 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:05:52.303426 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:05:52.303426 ip-10-0-140-194 kubenswrapper[2587]:  >
Sep 17 19:05:52.303426 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:05:52.303235    2587 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:05:52.303426 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:05:52.303426 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:05:52.303426 ip-10-0-140-194 kubenswrapper[2587]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm" podUID="8c23bc66-b192-4813-aa74-ea6166e422d8" containerName="main" containerID="cri-o://caf9a50a942116fb9a4b29c6d36c329db60f0de346232d7fb74a7f70f8fb9659"
Sep 17 19:05:52.303426 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.303293    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm" podUID="8c23bc66-b192-4813-aa74-ea6166e422d8" containerName="main" containerID="cri-o://caf9a50a942116fb9a4b29c6d36c329db60f0de346232d7fb74a7f70f8fb9659" gracePeriod=60
Sep 17 19:05:52.303426 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.303323    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"]
Sep 17 19:05:52.303763 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.303654    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf" podUID="2df6b1a7-573f-409d-b535-a8af086eba03" containerName="main" containerID="cri-o://71b7651902ae14ecc0314d94fbe6ce64cefbffb2870585c15b1990851db199a2" gracePeriod=30
Sep 17 19:05:52.351157 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.351109    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:52.423980 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.423930    2587 generic.go:356] "Generic (PLEG): container finished" podID="8c23bc66-b192-4813-aa74-ea6166e422d8" containerID="caf9a50a942116fb9a4b29c6d36c329db60f0de346232d7fb74a7f70f8fb9659" exitCode=0
Sep 17 19:05:52.424097 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.424005    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm" event={"ID":"8c23bc66-b192-4813-aa74-ea6166e422d8","Type":"ContainerDied","Data":"caf9a50a942116fb9a4b29c6d36c329db60f0de346232d7fb74a7f70f8fb9659"}
Sep 17 19:05:52.863640 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.863611    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:05:52.911587 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.911554    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df6b1a7-573f-409d-b535-a8af086eba03-kserve-provision-location\") pod \"2df6b1a7-573f-409d-b535-a8af086eba03\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") "
Sep 17 19:05:52.911759 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.911619    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2df6b1a7-573f-409d-b535-a8af086eba03-cabundle-cert\") pod \"2df6b1a7-573f-409d-b535-a8af086eba03\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") "
Sep 17 19:05:52.911759 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.911665    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df6b1a7-573f-409d-b535-a8af086eba03-tls-certs\") pod \"2df6b1a7-573f-409d-b535-a8af086eba03\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") "
Sep 17 19:05:52.911759 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.911694    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df6b1a7-573f-409d-b535-a8af086eba03-tmp-dir\") pod \"2df6b1a7-573f-409d-b535-a8af086eba03\" (UID: \"2df6b1a7-573f-409d-b535-a8af086eba03\") "
Sep 17 19:05:52.911980 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.911923    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df6b1a7-573f-409d-b535-a8af086eba03-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2df6b1a7-573f-409d-b535-a8af086eba03" (UID: "2df6b1a7-573f-409d-b535-a8af086eba03"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:05:52.912225 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.912201    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df6b1a7-573f-409d-b535-a8af086eba03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2df6b1a7-573f-409d-b535-a8af086eba03" (UID: "2df6b1a7-573f-409d-b535-a8af086eba03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:52.914237 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.914214    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df6b1a7-573f-409d-b535-a8af086eba03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2df6b1a7-573f-409d-b535-a8af086eba03" (UID: "2df6b1a7-573f-409d-b535-a8af086eba03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:05:52.936130 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:52.936040    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df6b1a7-573f-409d-b535-a8af086eba03-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2df6b1a7-573f-409d-b535-a8af086eba03" (UID: "2df6b1a7-573f-409d-b535-a8af086eba03"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:53.012652 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.012620    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2df6b1a7-573f-409d-b535-a8af086eba03-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:05:53.012652 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.012651    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2df6b1a7-573f-409d-b535-a8af086eba03-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:05:53.012652 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.012660    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df6b1a7-573f-409d-b535-a8af086eba03-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:05:53.012875 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.012669    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df6b1a7-573f-409d-b535-a8af086eba03-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:05:53.043435 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.043407    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:05:53.113766 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.113726    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwqfb\" (UniqueName: \"kubernetes.io/projected/8c23bc66-b192-4813-aa74-ea6166e422d8-kube-api-access-zwqfb\") pod \"8c23bc66-b192-4813-aa74-ea6166e422d8\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") "
Sep 17 19:05:53.113997 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.113773    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-home\") pod \"8c23bc66-b192-4813-aa74-ea6166e422d8\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") "
Sep 17 19:05:53.113997 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.113803    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-kserve-provision-location\") pod \"8c23bc66-b192-4813-aa74-ea6166e422d8\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") "
Sep 17 19:05:53.113997 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.113845    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c23bc66-b192-4813-aa74-ea6166e422d8-cabundle-cert\") pod \"8c23bc66-b192-4813-aa74-ea6166e422d8\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") "
Sep 17 19:05:53.113997 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.113899    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-tmp-dir\") pod \"8c23bc66-b192-4813-aa74-ea6166e422d8\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") "
Sep 17 19:05:53.113997 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.113970    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-model-cache\") pod \"8c23bc66-b192-4813-aa74-ea6166e422d8\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") "
Sep 17 19:05:53.114267 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.114000    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c23bc66-b192-4813-aa74-ea6166e422d8-tls-certs\") pod \"8c23bc66-b192-4813-aa74-ea6166e422d8\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") "
Sep 17 19:05:53.114267 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.114042    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-dshm\") pod \"8c23bc66-b192-4813-aa74-ea6166e422d8\" (UID: \"8c23bc66-b192-4813-aa74-ea6166e422d8\") "
Sep 17 19:05:53.114267 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.114114    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-home" (OuterVolumeSpecName: "home") pod "8c23bc66-b192-4813-aa74-ea6166e422d8" (UID: "8c23bc66-b192-4813-aa74-ea6166e422d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:53.114267 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.114201    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c23bc66-b192-4813-aa74-ea6166e422d8" (UID: "8c23bc66-b192-4813-aa74-ea6166e422d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:53.114477 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.114282    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c23bc66-b192-4813-aa74-ea6166e422d8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c23bc66-b192-4813-aa74-ea6166e422d8" (UID: "8c23bc66-b192-4813-aa74-ea6166e422d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:05:53.114477 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.114300    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c23bc66-b192-4813-aa74-ea6166e422d8" (UID: "8c23bc66-b192-4813-aa74-ea6166e422d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:53.114477 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.114313    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:05:53.114477 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.114329    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:05:53.116254 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.116228    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c23bc66-b192-4813-aa74-ea6166e422d8-kube-api-access-zwqfb" (OuterVolumeSpecName: "kube-api-access-zwqfb") pod "8c23bc66-b192-4813-aa74-ea6166e422d8" (UID: "8c23bc66-b192-4813-aa74-ea6166e422d8"). InnerVolumeSpecName "kube-api-access-zwqfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:05:53.116408 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.116392    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c23bc66-b192-4813-aa74-ea6166e422d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c23bc66-b192-4813-aa74-ea6166e422d8" (UID: "8c23bc66-b192-4813-aa74-ea6166e422d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:05:53.116460 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.116442    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-dshm" (OuterVolumeSpecName: "dshm") pod "8c23bc66-b192-4813-aa74-ea6166e422d8" (UID: "8c23bc66-b192-4813-aa74-ea6166e422d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:53.135800 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.135762    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c23bc66-b192-4813-aa74-ea6166e422d8" (UID: "8c23bc66-b192-4813-aa74-ea6166e422d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:53.214982 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.214863    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:05:53.214982 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.214897    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c23bc66-b192-4813-aa74-ea6166e422d8-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:05:53.214982 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.214906    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:05:53.214982 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.214915    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwqfb\" (UniqueName: \"kubernetes.io/projected/8c23bc66-b192-4813-aa74-ea6166e422d8-kube-api-access-zwqfb\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:05:53.214982 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.214924    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c23bc66-b192-4813-aa74-ea6166e422d8-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:05:53.214982 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.214933    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c23bc66-b192-4813-aa74-ea6166e422d8-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:05:53.432745 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.432714    2587 generic.go:356] "Generic (PLEG): container finished" podID="2df6b1a7-573f-409d-b535-a8af086eba03" containerID="71b7651902ae14ecc0314d94fbe6ce64cefbffb2870585c15b1990851db199a2" exitCode=0
Sep 17 19:05:53.433239 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.432766    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"
Sep 17 19:05:53.433239 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.432802    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf" event={"ID":"2df6b1a7-573f-409d-b535-a8af086eba03","Type":"ContainerDied","Data":"71b7651902ae14ecc0314d94fbe6ce64cefbffb2870585c15b1990851db199a2"}
Sep 17 19:05:53.433239 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.432855    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf" event={"ID":"2df6b1a7-573f-409d-b535-a8af086eba03","Type":"ContainerDied","Data":"c4c6fabe7ac363620738f585fa2c010253850553dc30bbecd5f33b9c87acb8cd"}
Sep 17 19:05:53.433239 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.432874    2587 scope.go:117] "RemoveContainer" containerID="71b7651902ae14ecc0314d94fbe6ce64cefbffb2870585c15b1990851db199a2"
Sep 17 19:05:53.434919 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.434886    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"
Sep 17 19:05:53.434919 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.434891    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm" event={"ID":"8c23bc66-b192-4813-aa74-ea6166e422d8","Type":"ContainerDied","Data":"0e4f534d04bdfe8b1690a2def2ec8ca1859d3e4a078ab93a796544aac121b3d1"}
Sep 17 19:05:53.443419 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.443404    2587 scope.go:117] "RemoveContainer" containerID="15908e9f94c0f106228bdfcea6a2b3ddb9c4b74d7744bd4e53368940a9c8bb5c"
Sep 17 19:05:53.451868 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.451851    2587 scope.go:117] "RemoveContainer" containerID="71b7651902ae14ecc0314d94fbe6ce64cefbffb2870585c15b1990851db199a2"
Sep 17 19:05:53.452198 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:05:53.452178    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b7651902ae14ecc0314d94fbe6ce64cefbffb2870585c15b1990851db199a2\": container with ID starting with 71b7651902ae14ecc0314d94fbe6ce64cefbffb2870585c15b1990851db199a2 not found: ID does not exist" containerID="71b7651902ae14ecc0314d94fbe6ce64cefbffb2870585c15b1990851db199a2"
Sep 17 19:05:53.452264 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.452207    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b7651902ae14ecc0314d94fbe6ce64cefbffb2870585c15b1990851db199a2"} err="failed to get container status \"71b7651902ae14ecc0314d94fbe6ce64cefbffb2870585c15b1990851db199a2\": rpc error: code = NotFound desc = could not find container \"71b7651902ae14ecc0314d94fbe6ce64cefbffb2870585c15b1990851db199a2\": container with ID starting with 71b7651902ae14ecc0314d94fbe6ce64cefbffb2870585c15b1990851db199a2 not found: ID does not exist"
Sep 17 19:05:53.452264 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.452224    2587 scope.go:117] "RemoveContainer" containerID="15908e9f94c0f106228bdfcea6a2b3ddb9c4b74d7744bd4e53368940a9c8bb5c"
Sep 17 19:05:53.452511 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:05:53.452491    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15908e9f94c0f106228bdfcea6a2b3ddb9c4b74d7744bd4e53368940a9c8bb5c\": container with ID starting with 15908e9f94c0f106228bdfcea6a2b3ddb9c4b74d7744bd4e53368940a9c8bb5c not found: ID does not exist" containerID="15908e9f94c0f106228bdfcea6a2b3ddb9c4b74d7744bd4e53368940a9c8bb5c"
Sep 17 19:05:53.452563 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.452517    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15908e9f94c0f106228bdfcea6a2b3ddb9c4b74d7744bd4e53368940a9c8bb5c"} err="failed to get container status \"15908e9f94c0f106228bdfcea6a2b3ddb9c4b74d7744bd4e53368940a9c8bb5c\": rpc error: code = NotFound desc = could not find container \"15908e9f94c0f106228bdfcea6a2b3ddb9c4b74d7744bd4e53368940a9c8bb5c\": container with ID starting with 15908e9f94c0f106228bdfcea6a2b3ddb9c4b74d7744bd4e53368940a9c8bb5c not found: ID does not exist"
Sep 17 19:05:53.452563 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.452533    2587 scope.go:117] "RemoveContainer" containerID="caf9a50a942116fb9a4b29c6d36c329db60f0de346232d7fb74a7f70f8fb9659"
Sep 17 19:05:53.460245 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.460221    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"]
Sep 17 19:05:53.460746 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.460722    2587 scope.go:117] "RemoveContainer" containerID="042acf105c62b1090fe150a7ea613b74d5089ffde79eddd3e60bdc6012645946"
Sep 17 19:05:53.465726 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.465676    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-64876b6f79-shsgf"]
Sep 17 19:05:53.476881 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.476857    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"]
Sep 17 19:05:53.482687 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:53.482666    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-87bfb5b97-46kqm"]
Sep 17 19:05:54.258898 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:54.258866    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2df6b1a7-573f-409d-b535-a8af086eba03" path="/var/lib/kubelet/pods/2df6b1a7-573f-409d-b535-a8af086eba03/volumes"
Sep 17 19:05:54.259371 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:05:54.259356    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c23bc66-b192-4813-aa74-ea6166e422d8" path="/var/lib/kubelet/pods/8c23bc66-b192-4813-aa74-ea6166e422d8/volumes"
Sep 17 19:06:40.101177 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:40.101138    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:40.302957 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:40.302909    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:47.031770 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:47.031735    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:53.114269 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:53.114230    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:53.196205 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:53.196168    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:58.698648 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.698608    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"]
Sep 17 19:06:58.699059 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.698903    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c23bc66-b192-4813-aa74-ea6166e422d8" containerName="storage-initializer"
Sep 17 19:06:58.699059 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.698914    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c23bc66-b192-4813-aa74-ea6166e422d8" containerName="storage-initializer"
Sep 17 19:06:58.699059 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.698931    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df6b1a7-573f-409d-b535-a8af086eba03" containerName="storage-initializer"
Sep 17 19:06:58.699059 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.698962    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df6b1a7-573f-409d-b535-a8af086eba03" containerName="storage-initializer"
Sep 17 19:06:58.699059 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.698971    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df6b1a7-573f-409d-b535-a8af086eba03" containerName="main"
Sep 17 19:06:58.699059 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.698976    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df6b1a7-573f-409d-b535-a8af086eba03" containerName="main"
Sep 17 19:06:58.699059 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.698991    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c23bc66-b192-4813-aa74-ea6166e422d8" containerName="main"
Sep 17 19:06:58.699059 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.698996    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c23bc66-b192-4813-aa74-ea6166e422d8" containerName="main"
Sep 17 19:06:58.699059 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.699044    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="2df6b1a7-573f-409d-b535-a8af086eba03" containerName="main"
Sep 17 19:06:58.699059 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.699053    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c23bc66-b192-4813-aa74-ea6166e422d8" containerName="main"
Sep 17 19:06:58.702255 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.702234    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.705093 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.705070    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 19:06:58.706116 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.706086    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-5czsv\""
Sep 17 19:06:58.706223 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.706133    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 19:06:58.706223 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.706133    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 19:06:58.717081 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.717056    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"]
Sep 17 19:06:58.723395 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.723364    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:58.756148 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.756117    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.756359 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.756166    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgd8b\" (UniqueName: \"kubernetes.io/projected/8198373f-c26f-462a-80ec-87f00272e8f3-kube-api-access-jgd8b\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.756359 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.756186    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.756359 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.756280    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.756359 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.756333    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8198373f-c26f-462a-80ec-87f00272e8f3-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.756536 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.756375    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.856820 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.856776    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.857050 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.856828    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgd8b\" (UniqueName: \"kubernetes.io/projected/8198373f-c26f-462a-80ec-87f00272e8f3-kube-api-access-jgd8b\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.857050 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.856856    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.857050 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.856885    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.857050 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.856916    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8198373f-c26f-462a-80ec-87f00272e8f3-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.857288 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.857093    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.857288 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.857254    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.857383 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.857304    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.857420 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.857406    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.859301 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.859279    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.859597 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.859578    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8198373f-c26f-462a-80ec-87f00272e8f3-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.871087 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.871059    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgd8b\" (UniqueName: \"kubernetes.io/projected/8198373f-c26f-462a-80ec-87f00272e8f3-kube-api-access-jgd8b\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:58.970283 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.970199    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"]
Sep 17 19:06:58.973462 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.973443    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"
Sep 17 19:06:58.976387 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:58.976364    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-wc6sd\""
Sep 17 19:06:59.006359 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.006322    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"]
Sep 17 19:06:59.012104 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.012053    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:06:59.058602 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.058563    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p6hz\" (UniqueName: \"kubernetes.io/projected/6161c766-0855-4140-b755-e0950f3317b8-kube-api-access-6p6hz\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf\" (UID: \"6161c766-0855-4140-b755-e0950f3317b8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"
Sep 17 19:06:59.058819 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.058633    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6161c766-0855-4140-b755-e0950f3317b8-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf\" (UID: \"6161c766-0855-4140-b755-e0950f3317b8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"
Sep 17 19:06:59.065172 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.064996    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:59.159450 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.159412    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6161c766-0855-4140-b755-e0950f3317b8-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf\" (UID: \"6161c766-0855-4140-b755-e0950f3317b8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"
Sep 17 19:06:59.159642 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.159500    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p6hz\" (UniqueName: \"kubernetes.io/projected/6161c766-0855-4140-b755-e0950f3317b8-kube-api-access-6p6hz\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf\" (UID: \"6161c766-0855-4140-b755-e0950f3317b8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"
Sep 17 19:06:59.162183 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.162164    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6161c766-0855-4140-b755-e0950f3317b8-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf\" (UID: \"6161c766-0855-4140-b755-e0950f3317b8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"
Sep 17 19:06:59.169491 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.169469    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p6hz\" (UniqueName: \"kubernetes.io/projected/6161c766-0855-4140-b755-e0950f3317b8-kube-api-access-6p6hz\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf\" (UID: \"6161c766-0855-4140-b755-e0950f3317b8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"
Sep 17 19:06:59.284110 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.284059    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"
Sep 17 19:06:59.294265 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.294240    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"]
Sep 17 19:06:59.296613 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:06:59.296584    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8198373f_c26f_462a_80ec_87f00272e8f3.slice/crio-e21e06d302db5d1c773487cf2fd1a5d8a7dff6f81f6471d286ea19e3d49ab635 WatchSource:0}: Error finding container e21e06d302db5d1c773487cf2fd1a5d8a7dff6f81f6471d286ea19e3d49ab635: Status 404 returned error can't find the container with id e21e06d302db5d1c773487cf2fd1a5d8a7dff6f81f6471d286ea19e3d49ab635
Sep 17 19:06:59.564404 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.564377    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"]
Sep 17 19:06:59.566008 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:06:59.565976    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6161c766_0855_4140_b755_e0950f3317b8.slice/crio-5994749f6cbd64b87c0a53cd7175833e4f2ed9c09c8a9e5064bc3a6eeafbc5ac WatchSource:0}: Error finding container 5994749f6cbd64b87c0a53cd7175833e4f2ed9c09c8a9e5064bc3a6eeafbc5ac: Status 404 returned error can't find the container with id 5994749f6cbd64b87c0a53cd7175833e4f2ed9c09c8a9e5064bc3a6eeafbc5ac
Sep 17 19:06:59.752490 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.752454    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf" event={"ID":"6161c766-0855-4140-b755-e0950f3317b8","Type":"ContainerStarted","Data":"42877af6052ff061cbad4e5408a430ecf617ec207fecc6ce16c32cb9a97af9c4"}
Sep 17 19:06:59.752953 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.752520    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf" event={"ID":"6161c766-0855-4140-b755-e0950f3317b8","Type":"ContainerStarted","Data":"5994749f6cbd64b87c0a53cd7175833e4f2ed9c09c8a9e5064bc3a6eeafbc5ac"}
Sep 17 19:06:59.752953 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.752642    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"
Sep 17 19:06:59.753985 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.753961    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx" event={"ID":"8198373f-c26f-462a-80ec-87f00272e8f3","Type":"ContainerStarted","Data":"bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703"}
Sep 17 19:06:59.754075 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.754001    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx" event={"ID":"8198373f-c26f-462a-80ec-87f00272e8f3","Type":"ContainerStarted","Data":"e21e06d302db5d1c773487cf2fd1a5d8a7dff6f81f6471d286ea19e3d49ab635"}
Sep 17 19:06:59.778905 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.778846    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf" podStartSLOduration=1.778831348 podStartE2EDuration="1.778831348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:59.777242008 +0000 UTC m=+1671.656210764" watchObservedRunningTime="2026-09-17 19:06:59.778831348 +0000 UTC m=+1671.657800104"
Sep 17 19:06:59.802286 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:06:59.802229    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx" podStartSLOduration=1.8022126950000001 podStartE2EDuration="1.802212695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:59.800650273 +0000 UTC m=+1671.679619029" watchObservedRunningTime="2026-09-17 19:06:59.802212695 +0000 UTC m=+1671.681181522"
Sep 17 19:07:00.012437 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:00.012396    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:07:09.012963 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:09.012900    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:07:09.025798 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:09.025763    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:07:09.809433 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:09.809400    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:07:10.336571 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:10.336536    2587 scope.go:117] "RemoveContainer" containerID="c3277fbb56273fa9eb9cf82f56522f172c709c85c676caa446097c0cd8f3ab37"
Sep 17 19:07:10.345119 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:10.345100    2587 scope.go:117] "RemoveContainer" containerID="775df419f4e7b2d55f2db3b696761ba9e4187b09a29a87c3606a12912cf3b854"
Sep 17 19:07:20.768373 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:20.768298    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:20.848663 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:20.848625    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:30.760656 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:30.760625    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"
Sep 17 19:07:48.152901 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:48.152863    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:52.550374 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.550331    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:52.567629 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.567599    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"]
Sep 17 19:07:52.572105 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.572075    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"]
Sep 17 19:07:52.580592 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:07:52.580558    2587 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:07:52.580592 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:07:52.580592 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:07:52.580592 ip-10-0-140-194 kubenswrapper[2587]:  > containerID="bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703" cmd=["/bin/sleep","15"]
Sep 17 19:07:52.580816 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.580617    2587 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx" message="" err=<
Sep 17 19:07:52.580816 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:07:52.580816 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:07:52.580816 ip-10-0-140-194 kubenswrapper[2587]:  >
Sep 17 19:07:52.580816 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:07:52.580634    2587 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:07:52.580816 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:07:52.580816 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:07:52.580816 ip-10-0-140-194 kubenswrapper[2587]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx" podUID="8198373f-c26f-462a-80ec-87f00272e8f3" containerName="main" containerID="cri-o://bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703"
Sep 17 19:07:52.580816 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.580677    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx" podUID="8198373f-c26f-462a-80ec-87f00272e8f3" containerName="main" containerID="cri-o://bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703" gracePeriod=60
Sep 17 19:07:52.647353 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.647315    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:52.799642 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.799592    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx" podUID="8198373f-c26f-462a-80ec-87f00272e8f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 17 19:07:52.931111 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.931088    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:07:52.987637 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.987602    2587 generic.go:356] "Generic (PLEG): container finished" podID="8198373f-c26f-462a-80ec-87f00272e8f3" containerID="bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703" exitCode=0
Sep 17 19:07:52.987836 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.987649    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx" event={"ID":"8198373f-c26f-462a-80ec-87f00272e8f3","Type":"ContainerDied","Data":"bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703"}
Sep 17 19:07:52.987836 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.987650    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"
Sep 17 19:07:52.987836 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.987691    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx" event={"ID":"8198373f-c26f-462a-80ec-87f00272e8f3","Type":"ContainerDied","Data":"e21e06d302db5d1c773487cf2fd1a5d8a7dff6f81f6471d286ea19e3d49ab635"}
Sep 17 19:07:52.987836 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.987714    2587 scope.go:117] "RemoveContainer" containerID="bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703"
Sep 17 19:07:52.996247 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.996230    2587 scope.go:117] "RemoveContainer" containerID="bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703"
Sep 17 19:07:52.996532 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:07:52.996516    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703\": container with ID starting with bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703 not found: ID does not exist" containerID="bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703"
Sep 17 19:07:52.996579 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:52.996540    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703"} err="failed to get container status \"bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703\": rpc error: code = NotFound desc = could not find container \"bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703\": container with ID starting with bd8b50c139c31c8ba36a28f3f9f7478d4993005bb7c3913913dfd52fed9dd703 not found: ID does not exist"
Sep 17 19:07:53.004790 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.004769    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-model-cache\") pod \"8198373f-c26f-462a-80ec-87f00272e8f3\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") "
Sep 17 19:07:53.004881 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.004815    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-tmp-dir\") pod \"8198373f-c26f-462a-80ec-87f00272e8f3\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") "
Sep 17 19:07:53.004881 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.004836    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgd8b\" (UniqueName: \"kubernetes.io/projected/8198373f-c26f-462a-80ec-87f00272e8f3-kube-api-access-jgd8b\") pod \"8198373f-c26f-462a-80ec-87f00272e8f3\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") "
Sep 17 19:07:53.004881 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.004873    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-dshm\") pod \"8198373f-c26f-462a-80ec-87f00272e8f3\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") "
Sep 17 19:07:53.005008 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.004898    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8198373f-c26f-462a-80ec-87f00272e8f3-tls-certs\") pod \"8198373f-c26f-462a-80ec-87f00272e8f3\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") "
Sep 17 19:07:53.005008 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.004915    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-home\") pod \"8198373f-c26f-462a-80ec-87f00272e8f3\" (UID: \"8198373f-c26f-462a-80ec-87f00272e8f3\") "
Sep 17 19:07:53.005255 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.005239    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-home" (OuterVolumeSpecName: "home") pod "8198373f-c26f-462a-80ec-87f00272e8f3" (UID: "8198373f-c26f-462a-80ec-87f00272e8f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:07:53.010173 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.008011    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8198373f-c26f-462a-80ec-87f00272e8f3" (UID: "8198373f-c26f-462a-80ec-87f00272e8f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:07:53.010173 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.008056    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "8198373f-c26f-462a-80ec-87f00272e8f3" (UID: "8198373f-c26f-462a-80ec-87f00272e8f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:07:53.014569 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.010701    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8198373f-c26f-462a-80ec-87f00272e8f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8198373f-c26f-462a-80ec-87f00272e8f3" (UID: "8198373f-c26f-462a-80ec-87f00272e8f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:07:53.015144 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.015111    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-dshm" (OuterVolumeSpecName: "dshm") pod "8198373f-c26f-462a-80ec-87f00272e8f3" (UID: "8198373f-c26f-462a-80ec-87f00272e8f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:07:53.015245 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.015147    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8198373f-c26f-462a-80ec-87f00272e8f3-kube-api-access-jgd8b" (OuterVolumeSpecName: "kube-api-access-jgd8b") pod "8198373f-c26f-462a-80ec-87f00272e8f3" (UID: "8198373f-c26f-462a-80ec-87f00272e8f3"). InnerVolumeSpecName "kube-api-access-jgd8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:07:53.105634 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.105524    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:07:53.105634 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.105598    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8198373f-c26f-462a-80ec-87f00272e8f3-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:07:53.105634 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.105614    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:07:53.105634 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.105625    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:07:53.105634 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.105636    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8198373f-c26f-462a-80ec-87f00272e8f3-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:07:53.105920 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.105648    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgd8b\" (UniqueName: \"kubernetes.io/projected/8198373f-c26f-462a-80ec-87f00272e8f3-kube-api-access-jgd8b\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:07:53.310369 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.310331    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"]
Sep 17 19:07:53.319915 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:53.319886    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-jzvdx"]
Sep 17 19:07:54.257691 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:54.257659    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8198373f-c26f-462a-80ec-87f00272e8f3" path="/var/lib/kubelet/pods/8198373f-c26f-462a-80ec-87f00272e8f3/volumes"
Sep 17 19:07:56.145570 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:56.145535    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:56.220081 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:07:56.220043    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:00.761432 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:00.761388    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf" podUID="6161c766-0855-4140-b755-e0950f3317b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:08:07.588017 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:07.587918    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf" podUID="6161c766-0855-4140-b755-e0950f3317b8" containerName="main" containerID="cri-o://42877af6052ff061cbad4e5408a430ecf617ec207fecc6ce16c32cb9a97af9c4" gracePeriod=45
Sep 17 19:08:10.761271 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:10.761235    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf" podUID="6161c766-0855-4140-b755-e0950f3317b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:08:20.760655 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:20.760609    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf" podUID="6161c766-0855-4140-b755-e0950f3317b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:08:20.761084 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:20.760716    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"
Sep 17 19:08:30.760855 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:30.760818    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf" podUID="6161c766-0855-4140-b755-e0950f3317b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:08:37.832932 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:37.832906    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"
Sep 17 19:08:37.876396 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:37.876310    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6161c766-0855-4140-b755-e0950f3317b8-tls-certs\") pod \"6161c766-0855-4140-b755-e0950f3317b8\" (UID: \"6161c766-0855-4140-b755-e0950f3317b8\") "
Sep 17 19:08:37.876396 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:37.876357    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p6hz\" (UniqueName: \"kubernetes.io/projected/6161c766-0855-4140-b755-e0950f3317b8-kube-api-access-6p6hz\") pod \"6161c766-0855-4140-b755-e0950f3317b8\" (UID: \"6161c766-0855-4140-b755-e0950f3317b8\") "
Sep 17 19:08:37.878594 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:37.878560    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6161c766-0855-4140-b755-e0950f3317b8-kube-api-access-6p6hz" (OuterVolumeSpecName: "kube-api-access-6p6hz") pod "6161c766-0855-4140-b755-e0950f3317b8" (UID: "6161c766-0855-4140-b755-e0950f3317b8"). InnerVolumeSpecName "kube-api-access-6p6hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:37.878594 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:37.878582    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6161c766-0855-4140-b755-e0950f3317b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6161c766-0855-4140-b755-e0950f3317b8" (UID: "6161c766-0855-4140-b755-e0950f3317b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:37.977655 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:37.977620    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6161c766-0855-4140-b755-e0950f3317b8-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:08:37.977655 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:37.977649    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p6hz\" (UniqueName: \"kubernetes.io/projected/6161c766-0855-4140-b755-e0950f3317b8-kube-api-access-6p6hz\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:08:38.186340 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:38.186246    2587 generic.go:356] "Generic (PLEG): container finished" podID="6161c766-0855-4140-b755-e0950f3317b8" containerID="42877af6052ff061cbad4e5408a430ecf617ec207fecc6ce16c32cb9a97af9c4" exitCode=0
Sep 17 19:08:38.186340 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:38.186293    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"
Sep 17 19:08:38.186551 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:38.186332    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf" event={"ID":"6161c766-0855-4140-b755-e0950f3317b8","Type":"ContainerDied","Data":"42877af6052ff061cbad4e5408a430ecf617ec207fecc6ce16c32cb9a97af9c4"}
Sep 17 19:08:38.186551 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:38.186396    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf" event={"ID":"6161c766-0855-4140-b755-e0950f3317b8","Type":"ContainerDied","Data":"5994749f6cbd64b87c0a53cd7175833e4f2ed9c09c8a9e5064bc3a6eeafbc5ac"}
Sep 17 19:08:38.186551 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:38.186423    2587 scope.go:117] "RemoveContainer" containerID="42877af6052ff061cbad4e5408a430ecf617ec207fecc6ce16c32cb9a97af9c4"
Sep 17 19:08:38.195640 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:38.195617    2587 scope.go:117] "RemoveContainer" containerID="42877af6052ff061cbad4e5408a430ecf617ec207fecc6ce16c32cb9a97af9c4"
Sep 17 19:08:38.195961 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:08:38.195922    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42877af6052ff061cbad4e5408a430ecf617ec207fecc6ce16c32cb9a97af9c4\": container with ID starting with 42877af6052ff061cbad4e5408a430ecf617ec207fecc6ce16c32cb9a97af9c4 not found: ID does not exist" containerID="42877af6052ff061cbad4e5408a430ecf617ec207fecc6ce16c32cb9a97af9c4"
Sep 17 19:08:38.196012 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:38.195975    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42877af6052ff061cbad4e5408a430ecf617ec207fecc6ce16c32cb9a97af9c4"} err="failed to get container status \"42877af6052ff061cbad4e5408a430ecf617ec207fecc6ce16c32cb9a97af9c4\": rpc error: code = NotFound desc = could not find container \"42877af6052ff061cbad4e5408a430ecf617ec207fecc6ce16c32cb9a97af9c4\": container with ID starting with 42877af6052ff061cbad4e5408a430ecf617ec207fecc6ce16c32cb9a97af9c4 not found: ID does not exist"
Sep 17 19:08:38.208831 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:38.208802    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"]
Sep 17 19:08:38.213631 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:38.213606    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schehp5wf"]
Sep 17 19:08:38.257821 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:38.257788    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6161c766-0855-4140-b755-e0950f3317b8" path="/var/lib/kubelet/pods/6161c766-0855-4140-b755-e0950f3317b8/volumes"
Sep 17 19:08:42.181254 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:42.181223    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:42.281788 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:42.281752    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:45.106146 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:45.106104    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:45.225255 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:45.225221    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:45.440710 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:45.440673    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:45.544069 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:45.544031    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:48.973671 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:48.973637    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"]
Sep 17 19:08:48.974098 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:48.974010    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8198373f-c26f-462a-80ec-87f00272e8f3" containerName="main"
Sep 17 19:08:48.974098 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:48.974023    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="8198373f-c26f-462a-80ec-87f00272e8f3" containerName="main"
Sep 17 19:08:48.974098 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:48.974033    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6161c766-0855-4140-b755-e0950f3317b8" containerName="main"
Sep 17 19:08:48.974098 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:48.974038    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="6161c766-0855-4140-b755-e0950f3317b8" containerName="main"
Sep 17 19:08:48.974098 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:48.974096    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="6161c766-0855-4140-b755-e0950f3317b8" containerName="main"
Sep 17 19:08:48.974283 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:48.974104    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="8198373f-c26f-462a-80ec-87f00272e8f3" containerName="main"
Sep 17 19:08:48.983078 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:48.983045    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:48.986343 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:48.986319    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 19:08:48.986770 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:48.986745    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"]
Sep 17 19:08:48.987294 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:48.987274    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 19:08:48.987405 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:48.987292    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-x5xqb\""
Sep 17 19:08:48.987405 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:48.987292    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 19:08:49.002466 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.002431    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:49.069667 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.069633    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.069856 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.069673    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.069856 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.069694    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmz9j\" (UniqueName: \"kubernetes.io/projected/37c6cc71-a216-4621-9525-d420e33e8795-kube-api-access-cmz9j\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.069856 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.069792    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.069856 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.069848    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c6cc71-a216-4621-9525-d420e33e8795-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.070057 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.069871    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.102403 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.102371    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:49.170840 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.170806    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c6cc71-a216-4621-9525-d420e33e8795-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.170840 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.170852    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.171162 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.170986    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.171162 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.171030    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.171162 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.171055    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmz9j\" (UniqueName: \"kubernetes.io/projected/37c6cc71-a216-4621-9525-d420e33e8795-kube-api-access-cmz9j\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.171162 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.171141    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.171383 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.171334    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.171383 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.171356    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.171464 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.171406    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.173411 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.173383    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.173666 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.173648    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c6cc71-a216-4621-9525-d420e33e8795-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.183387 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.183364    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmz9j\" (UniqueName: \"kubernetes.io/projected/37c6cc71-a216-4621-9525-d420e33e8795-kube-api-access-cmz9j\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.206994 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.206929    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:49.294799 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.294766    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:49.484732 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.484683    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:49.566774 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:49.566747    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"]
Sep 17 19:08:49.569075 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:08:49.569048    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c6cc71_a216_4621_9525_d420e33e8795.slice/crio-60c4c7674a42c07950e5d0da0d2f348b1c531d975e3bf21e21e6fc5d58a21401 WatchSource:0}: Error finding container 60c4c7674a42c07950e5d0da0d2f348b1c531d975e3bf21e21e6fc5d58a21401: Status 404 returned error can't find the container with id 60c4c7674a42c07950e5d0da0d2f348b1c531d975e3bf21e21e6fc5d58a21401
Sep 17 19:08:50.242586 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:50.242548    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc" event={"ID":"37c6cc71-a216-4621-9525-d420e33e8795","Type":"ContainerStarted","Data":"9700f01ac154e27cd72e19f27f19f3cd92e0c1eacf41be4825123a25e85d0ae0"}
Sep 17 19:08:50.243090 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:50.242596    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc" event={"ID":"37c6cc71-a216-4621-9525-d420e33e8795","Type":"ContainerStarted","Data":"60c4c7674a42c07950e5d0da0d2f348b1c531d975e3bf21e21e6fc5d58a21401"}
Sep 17 19:08:50.265802 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:50.265732    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc" podStartSLOduration=2.265716852 podStartE2EDuration="2.265716852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:50.263611025 +0000 UTC m=+1782.142579784" watchObservedRunningTime="2026-09-17 19:08:50.265716852 +0000 UTC m=+1782.144685608"
Sep 17 19:08:50.295822 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:50.295791    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:52.325689 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:52.325660    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:54.780876 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:54.780840    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:54.846727 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:54.846687    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:57.772091 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:57.772054    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:57.841931 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:57.841889    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:59.295672 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:59.295633    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:59.308400 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:59.308371    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:08:59.920538 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:08:59.920504    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:00.300371 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:00.300339    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:09:10.279407 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:10.279372    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:09:10.282227 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:10.282206    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:09:15.345763 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:15.345722    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:36.834589 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:36.834542    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:37.046653 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:37.046621    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"]
Sep 17 19:09:37.049986 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:37.049965    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"
Sep 17 19:09:37.052953 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:37.052908    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-wkzc2\""
Sep 17 19:09:37.053111 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:37.052958    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 19:09:37.067447 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:37.067414    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"]
Sep 17 19:09:37.089504 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:37.089420    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:37.094363 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:37.094337    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46zh2\" (UniqueName: \"kubernetes.io/projected/80bc64d2-64a1-4fd5-8929-4cf602d27f76-kube-api-access-46zh2\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl\" (UID: \"80bc64d2-64a1-4fd5-8929-4cf602d27f76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"
Sep 17 19:09:37.094482 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:37.094372    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80bc64d2-64a1-4fd5-8929-4cf602d27f76-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl\" (UID: \"80bc64d2-64a1-4fd5-8929-4cf602d27f76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"
Sep 17 19:09:37.195548 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:37.195502    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46zh2\" (UniqueName: \"kubernetes.io/projected/80bc64d2-64a1-4fd5-8929-4cf602d27f76-kube-api-access-46zh2\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl\" (UID: \"80bc64d2-64a1-4fd5-8929-4cf602d27f76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"
Sep 17 19:09:37.195548 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:37.195546    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80bc64d2-64a1-4fd5-8929-4cf602d27f76-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl\" (UID: \"80bc64d2-64a1-4fd5-8929-4cf602d27f76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"
Sep 17 19:09:37.198483 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:37.198452    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80bc64d2-64a1-4fd5-8929-4cf602d27f76-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl\" (UID: \"80bc64d2-64a1-4fd5-8929-4cf602d27f76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"
Sep 17 19:09:37.205325 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:37.205296    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46zh2\" (UniqueName: \"kubernetes.io/projected/80bc64d2-64a1-4fd5-8929-4cf602d27f76-kube-api-access-46zh2\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl\" (UID: \"80bc64d2-64a1-4fd5-8929-4cf602d27f76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"
Sep 17 19:09:37.361020 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:37.360895    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"
Sep 17 19:09:37.633960 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:37.633908    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"]
Sep 17 19:09:37.636622 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:09:37.636598    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80bc64d2_64a1_4fd5_8929_4cf602d27f76.slice/crio-b0f34392fb61624f298b10f695f2ee46f60eecc2d8fbb6764bec853d9796c736 WatchSource:0}: Error finding container b0f34392fb61624f298b10f695f2ee46f60eecc2d8fbb6764bec853d9796c736: Status 404 returned error can't find the container with id b0f34392fb61624f298b10f695f2ee46f60eecc2d8fbb6764bec853d9796c736
Sep 17 19:09:38.456012 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:38.455981    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl" event={"ID":"80bc64d2-64a1-4fd5-8929-4cf602d27f76","Type":"ContainerStarted","Data":"e33abe5765630f5309afb79fe7d0da7d6b2f4d85fc083edfa065e3e57dfd8fdc"}
Sep 17 19:09:38.456433 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:38.456043    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl" event={"ID":"80bc64d2-64a1-4fd5-8929-4cf602d27f76","Type":"ContainerStarted","Data":"b0f34392fb61624f298b10f695f2ee46f60eecc2d8fbb6764bec853d9796c736"}
Sep 17 19:09:38.456433 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:38.456070    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"
Sep 17 19:09:38.476923 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:38.476871    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl" podStartSLOduration=1.476855063 podStartE2EDuration="1.476855063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:38.475326531 +0000 UTC m=+1830.354295289" watchObservedRunningTime="2026-09-17 19:09:38.476855063 +0000 UTC m=+1830.355823830"
Sep 17 19:09:53.198319 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:53.198272    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:55.416584 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:55.416547    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:56.815434 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:56.815401    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:56.913058 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:09:56.913030    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:01.825605 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:01.825561    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:09.462306 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:09.462271    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"
Sep 17 19:10:23.109794 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:23.109705    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:24.431091 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:24.431039    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:27.695981 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:27.695916    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:27.920843 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:27.920790    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:30.673066 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:30.673030    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:38.871163 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:38.871117    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:45.407657 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:45.407616    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:45.409106 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:45.409081    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"]
Sep 17 19:10:45.492486 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:45.492447    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:49.391843 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:49.391803    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:49.462563 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:49.462520    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl" podUID="80bc64d2-64a1-4fd5-8929-4cf602d27f76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:10:54.235861 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:54.235817    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:54.322521 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:54.322472    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:59.462599 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:10:59.462556    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl" podUID="80bc64d2-64a1-4fd5-8929-4cf602d27f76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:11:00.429606 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:00.429568    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl" podUID="80bc64d2-64a1-4fd5-8929-4cf602d27f76" containerName="main" containerID="cri-o://e33abe5765630f5309afb79fe7d0da7d6b2f4d85fc083edfa065e3e57dfd8fdc" gracePeriod=45
Sep 17 19:11:09.461654 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:09.461609    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl" podUID="80bc64d2-64a1-4fd5-8929-4cf602d27f76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:11:09.462112 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:09.461739    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"
Sep 17 19:11:19.462510 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:19.462460    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl" podUID="80bc64d2-64a1-4fd5-8929-4cf602d27f76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:11:21.679114 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:21.679070    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:21.778059 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:21.778018    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:29.462095 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:29.462050    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl" podUID="80bc64d2-64a1-4fd5-8929-4cf602d27f76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:11:30.690500 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.690406    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"
Sep 17 19:11:30.755684 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.755648    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46zh2\" (UniqueName: \"kubernetes.io/projected/80bc64d2-64a1-4fd5-8929-4cf602d27f76-kube-api-access-46zh2\") pod \"80bc64d2-64a1-4fd5-8929-4cf602d27f76\" (UID: \"80bc64d2-64a1-4fd5-8929-4cf602d27f76\") "
Sep 17 19:11:30.755891 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.755707    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80bc64d2-64a1-4fd5-8929-4cf602d27f76-tls-certs\") pod \"80bc64d2-64a1-4fd5-8929-4cf602d27f76\" (UID: \"80bc64d2-64a1-4fd5-8929-4cf602d27f76\") "
Sep 17 19:11:30.758249 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.758206    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80bc64d2-64a1-4fd5-8929-4cf602d27f76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80bc64d2-64a1-4fd5-8929-4cf602d27f76" (UID: "80bc64d2-64a1-4fd5-8929-4cf602d27f76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:11:30.758374 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.758248    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bc64d2-64a1-4fd5-8929-4cf602d27f76-kube-api-access-46zh2" (OuterVolumeSpecName: "kube-api-access-46zh2") pod "80bc64d2-64a1-4fd5-8929-4cf602d27f76" (UID: "80bc64d2-64a1-4fd5-8929-4cf602d27f76"). InnerVolumeSpecName "kube-api-access-46zh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:11:30.856994 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.856923    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46zh2\" (UniqueName: \"kubernetes.io/projected/80bc64d2-64a1-4fd5-8929-4cf602d27f76-kube-api-access-46zh2\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:11:30.856994 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.856989    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80bc64d2-64a1-4fd5-8929-4cf602d27f76-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:11:30.956336 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.956244    2587 generic.go:356] "Generic (PLEG): container finished" podID="80bc64d2-64a1-4fd5-8929-4cf602d27f76" containerID="e33abe5765630f5309afb79fe7d0da7d6b2f4d85fc083edfa065e3e57dfd8fdc" exitCode=0
Sep 17 19:11:30.956336 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.956290    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"
Sep 17 19:11:30.956336 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.956302    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl" event={"ID":"80bc64d2-64a1-4fd5-8929-4cf602d27f76","Type":"ContainerDied","Data":"e33abe5765630f5309afb79fe7d0da7d6b2f4d85fc083edfa065e3e57dfd8fdc"}
Sep 17 19:11:30.956577 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.956371    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl" event={"ID":"80bc64d2-64a1-4fd5-8929-4cf602d27f76","Type":"ContainerDied","Data":"b0f34392fb61624f298b10f695f2ee46f60eecc2d8fbb6764bec853d9796c736"}
Sep 17 19:11:30.956577 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.956395    2587 scope.go:117] "RemoveContainer" containerID="e33abe5765630f5309afb79fe7d0da7d6b2f4d85fc083edfa065e3e57dfd8fdc"
Sep 17 19:11:30.966098 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.966077    2587 scope.go:117] "RemoveContainer" containerID="e33abe5765630f5309afb79fe7d0da7d6b2f4d85fc083edfa065e3e57dfd8fdc"
Sep 17 19:11:30.966443 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:11:30.966417    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e33abe5765630f5309afb79fe7d0da7d6b2f4d85fc083edfa065e3e57dfd8fdc\": container with ID starting with e33abe5765630f5309afb79fe7d0da7d6b2f4d85fc083edfa065e3e57dfd8fdc not found: ID does not exist" containerID="e33abe5765630f5309afb79fe7d0da7d6b2f4d85fc083edfa065e3e57dfd8fdc"
Sep 17 19:11:30.966506 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.966452    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e33abe5765630f5309afb79fe7d0da7d6b2f4d85fc083edfa065e3e57dfd8fdc"} err="failed to get container status \"e33abe5765630f5309afb79fe7d0da7d6b2f4d85fc083edfa065e3e57dfd8fdc\": rpc error: code = NotFound desc = could not find container \"e33abe5765630f5309afb79fe7d0da7d6b2f4d85fc083edfa065e3e57dfd8fdc\": container with ID starting with e33abe5765630f5309afb79fe7d0da7d6b2f4d85fc083edfa065e3e57dfd8fdc not found: ID does not exist"
Sep 17 19:11:30.978725 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.978685    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"]
Sep 17 19:11:30.983557 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:30.983527    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schek84wl"]
Sep 17 19:11:32.257798 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:32.257764    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80bc64d2-64a1-4fd5-8929-4cf602d27f76" path="/var/lib/kubelet/pods/80bc64d2-64a1-4fd5-8929-4cf602d27f76/volumes"
Sep 17 19:11:33.634680 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:33.634639    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:33.635209 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:33.634799    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"]
Sep 17 19:11:33.644166 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:11:33.644125    2587 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:11:33.644166 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:11:33.644166 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:11:33.644166 ip-10-0-140-194 kubenswrapper[2587]:  > containerID="9700f01ac154e27cd72e19f27f19f3cd92e0c1eacf41be4825123a25e85d0ae0" cmd=["/bin/sleep","15"]
Sep 17 19:11:33.644411 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:33.644178    2587 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc" message="" err=<
Sep 17 19:11:33.644411 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:11:33.644411 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:11:33.644411 ip-10-0-140-194 kubenswrapper[2587]:  >
Sep 17 19:11:33.644411 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:11:33.644191    2587 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:11:33.644411 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:11:33.644411 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:11:33.644411 ip-10-0-140-194 kubenswrapper[2587]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc" podUID="37c6cc71-a216-4621-9525-d420e33e8795" containerName="main" containerID="cri-o://9700f01ac154e27cd72e19f27f19f3cd92e0c1eacf41be4825123a25e85d0ae0"
Sep 17 19:11:33.644411 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:33.644239    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc" podUID="37c6cc71-a216-4621-9525-d420e33e8795" containerName="main" containerID="cri-o://9700f01ac154e27cd72e19f27f19f3cd92e0c1eacf41be4825123a25e85d0ae0" gracePeriod=60
Sep 17 19:11:33.712456 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:33.712412    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:33.976250 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:33.976215    2587 generic.go:356] "Generic (PLEG): container finished" podID="37c6cc71-a216-4621-9525-d420e33e8795" containerID="9700f01ac154e27cd72e19f27f19f3cd92e0c1eacf41be4825123a25e85d0ae0" exitCode=0
Sep 17 19:11:33.976385 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:33.976283    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc" event={"ID":"37c6cc71-a216-4621-9525-d420e33e8795","Type":"ContainerDied","Data":"9700f01ac154e27cd72e19f27f19f3cd92e0c1eacf41be4825123a25e85d0ae0"}
Sep 17 19:11:33.995884 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:33.995849    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:11:34.084507 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.084466    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-home\") pod \"37c6cc71-a216-4621-9525-d420e33e8795\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") "
Sep 17 19:11:34.084507 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.084507    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-model-cache\") pod \"37c6cc71-a216-4621-9525-d420e33e8795\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") "
Sep 17 19:11:34.084783 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.084646    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c6cc71-a216-4621-9525-d420e33e8795-tls-certs\") pod \"37c6cc71-a216-4621-9525-d420e33e8795\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") "
Sep 17 19:11:34.084783 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.084699    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-dshm\") pod \"37c6cc71-a216-4621-9525-d420e33e8795\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") "
Sep 17 19:11:34.084783 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.084743    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmz9j\" (UniqueName: \"kubernetes.io/projected/37c6cc71-a216-4621-9525-d420e33e8795-kube-api-access-cmz9j\") pod \"37c6cc71-a216-4621-9525-d420e33e8795\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") "
Sep 17 19:11:34.084988 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.084778    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-home" (OuterVolumeSpecName: "home") pod "37c6cc71-a216-4621-9525-d420e33e8795" (UID: "37c6cc71-a216-4621-9525-d420e33e8795"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:11:34.084988 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.084796    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-model-cache" (OuterVolumeSpecName: "model-cache") pod "37c6cc71-a216-4621-9525-d420e33e8795" (UID: "37c6cc71-a216-4621-9525-d420e33e8795"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:11:34.084988 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.084803    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-tmp-dir\") pod \"37c6cc71-a216-4621-9525-d420e33e8795\" (UID: \"37c6cc71-a216-4621-9525-d420e33e8795\") "
Sep 17 19:11:34.085147 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.085072    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37c6cc71-a216-4621-9525-d420e33e8795" (UID: "37c6cc71-a216-4621-9525-d420e33e8795"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:11:34.085196 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.085140    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:11:34.085196 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.085163    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:11:34.087094 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.087073    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c6cc71-a216-4621-9525-d420e33e8795-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37c6cc71-a216-4621-9525-d420e33e8795" (UID: "37c6cc71-a216-4621-9525-d420e33e8795"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:11:34.087184 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.087146    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-dshm" (OuterVolumeSpecName: "dshm") pod "37c6cc71-a216-4621-9525-d420e33e8795" (UID: "37c6cc71-a216-4621-9525-d420e33e8795"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:11:34.087184 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.087160    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c6cc71-a216-4621-9525-d420e33e8795-kube-api-access-cmz9j" (OuterVolumeSpecName: "kube-api-access-cmz9j") pod "37c6cc71-a216-4621-9525-d420e33e8795" (UID: "37c6cc71-a216-4621-9525-d420e33e8795"). InnerVolumeSpecName "kube-api-access-cmz9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:11:34.185892 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.185796    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmz9j\" (UniqueName: \"kubernetes.io/projected/37c6cc71-a216-4621-9525-d420e33e8795-kube-api-access-cmz9j\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:11:34.185892 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.185832    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:11:34.185892 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.185843    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c6cc71-a216-4621-9525-d420e33e8795-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:11:34.185892 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.185852    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37c6cc71-a216-4621-9525-d420e33e8795-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:11:34.982721 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.982692    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"
Sep 17 19:11:34.983276 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.982689    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc" event={"ID":"37c6cc71-a216-4621-9525-d420e33e8795","Type":"ContainerDied","Data":"60c4c7674a42c07950e5d0da0d2f348b1c531d975e3bf21e21e6fc5d58a21401"}
Sep 17 19:11:34.983276 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:34.982837    2587 scope.go:117] "RemoveContainer" containerID="9700f01ac154e27cd72e19f27f19f3cd92e0c1eacf41be4825123a25e85d0ae0"
Sep 17 19:11:35.000562 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:35.000527    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"]
Sep 17 19:11:35.004043 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:35.004014    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-sh2qc"]
Sep 17 19:11:36.258083 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:36.258043    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37c6cc71-a216-4621-9525-d420e33e8795" path="/var/lib/kubelet/pods/37c6cc71-a216-4621-9525-d420e33e8795/volumes"
Sep 17 19:11:40.427987 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:11:40.427927    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:23.704140 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:23.704106    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:23.801296 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:23.801254    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:29.495815 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.495772    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"]
Sep 17 19:12:29.496254 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.496091    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37c6cc71-a216-4621-9525-d420e33e8795" containerName="main"
Sep 17 19:12:29.496254 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.496103    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="37c6cc71-a216-4621-9525-d420e33e8795" containerName="main"
Sep 17 19:12:29.496254 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.496127    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80bc64d2-64a1-4fd5-8929-4cf602d27f76" containerName="main"
Sep 17 19:12:29.496254 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.496132    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="80bc64d2-64a1-4fd5-8929-4cf602d27f76" containerName="main"
Sep 17 19:12:29.496254 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.496187    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="37c6cc71-a216-4621-9525-d420e33e8795" containerName="main"
Sep 17 19:12:29.496254 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.496196    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="80bc64d2-64a1-4fd5-8929-4cf602d27f76" containerName="main"
Sep 17 19:12:29.500505 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.500478    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.504383 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.504350    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-x7zbj\""
Sep 17 19:12:29.504558 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.504361    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 19:12:29.504558 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.504420    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 19:12:29.504558 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.504412    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 19:12:29.509756 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.509731    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"]
Sep 17 19:12:29.524747 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.524705    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:29.556401 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.556361    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.556401 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.556402    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.556692 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.556426    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1180168-c501-4286-b7ed-14b8e394713c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.556692 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.556528    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.556692 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.556583    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s65f7\" (UniqueName: \"kubernetes.io/projected/e1180168-c501-4286-b7ed-14b8e394713c-kube-api-access-s65f7\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.556692 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.556665    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.657714 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.657670    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.657714 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.657716    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.657987 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.657738    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1180168-c501-4286-b7ed-14b8e394713c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.657987 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.657758    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.657987 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.657787    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s65f7\" (UniqueName: \"kubernetes.io/projected/e1180168-c501-4286-b7ed-14b8e394713c-kube-api-access-s65f7\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.657987 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.657858    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.658256 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.658229    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.658375 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.658295    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.658375 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.658317    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.660347 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.660317    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.660655 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.660632    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1180168-c501-4286-b7ed-14b8e394713c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.673650 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.673622    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s65f7\" (UniqueName: \"kubernetes.io/projected/e1180168-c501-4286-b7ed-14b8e394713c-kube-api-access-s65f7\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.812214 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.812039    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:29.858209 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:29.858162    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:30.088531 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:30.088501    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"]
Sep 17 19:12:30.090735 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:12:30.090703    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1180168_c501_4286_b7ed_14b8e394713c.slice/crio-bbf34cfc072130dbaf031b54b530e05f708cc8eae801f944207a9e79ba2d40a5 WatchSource:0}: Error finding container bbf34cfc072130dbaf031b54b530e05f708cc8eae801f944207a9e79ba2d40a5: Status 404 returned error can't find the container with id bbf34cfc072130dbaf031b54b530e05f708cc8eae801f944207a9e79ba2d40a5
Sep 17 19:12:30.227387 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:30.227349    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s" event={"ID":"e1180168-c501-4286-b7ed-14b8e394713c","Type":"ContainerStarted","Data":"962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8"}
Sep 17 19:12:30.227586 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:30.227407    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s" event={"ID":"e1180168-c501-4286-b7ed-14b8e394713c","Type":"ContainerStarted","Data":"bbf34cfc072130dbaf031b54b530e05f708cc8eae801f944207a9e79ba2d40a5"}
Sep 17 19:12:30.247831 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:30.247775    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s" podStartSLOduration=1.247758318 podStartE2EDuration="1.247758318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:12:30.246558414 +0000 UTC m=+2002.125527173" watchObservedRunningTime="2026-09-17 19:12:30.247758318 +0000 UTC m=+2002.126727131"
Sep 17 19:12:30.812479 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:30.812420    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:31.113463 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:31.113375    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:36.695527 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:36.695486    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:36.771499 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:36.771465    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:39.812851 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:39.812812    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:39.825602 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:39.825569    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:40.283331 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:40.283299    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:12:41.689461 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:41.689429    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:48.753847 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:48.753812    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:48.815030 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:48.814991    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:54.963380 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:54.963347    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:55.064553 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:12:55.064517    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:10.616875 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:10.616835    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:10.843174 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:10.843111    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:34.038339 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:34.038247    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:38.833650 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:38.833614    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:38.928543 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:38.928502    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:41.302893 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.302849    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"]
Sep 17 19:13:41.306526 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.306502    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.309554 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.309521    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 19:13:41.309554 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.309540    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 19:13:41.309554 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.309554    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-vvw4j\""
Sep 17 19:13:41.309844 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.309525    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 19:13:41.337274 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.337229    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"]
Sep 17 19:13:41.344502 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.344460    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:41.377356 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.377320    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.377356 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.377371    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.377625 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.377394    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.377625 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.377423    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.377625 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.377443    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74b800d0-0a51-4383-ac5a-91863ce015d6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.377625 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.377463    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.377625 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.377522    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpwqs\" (UniqueName: \"kubernetes.io/projected/74b800d0-0a51-4383-ac5a-91863ce015d6-kube-api-access-jpwqs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.478731 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.478697    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.478731 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.478734    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.478990 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.478752    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.478990 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.478768    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74b800d0-0a51-4383-ac5a-91863ce015d6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.478990 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.478847    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.478990 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.478906    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpwqs\" (UniqueName: \"kubernetes.io/projected/74b800d0-0a51-4383-ac5a-91863ce015d6-kube-api-access-jpwqs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.479211 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.478994    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.479351 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.479310    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.479419 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.479323    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.479419 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.479385    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.479493 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.479431    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.481383 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.481351    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.481643 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.481625    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74b800d0-0a51-4383-ac5a-91863ce015d6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.495311 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.495275    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpwqs\" (UniqueName: \"kubernetes.io/projected/74b800d0-0a51-4383-ac5a-91863ce015d6-kube-api-access-jpwqs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.617518 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.617423    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:41.695105 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.695073    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:41.909663 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.909636    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"]
Sep 17 19:13:41.946897 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:41.946876    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:13:42.553953 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:42.553896    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" event={"ID":"74b800d0-0a51-4383-ac5a-91863ce015d6","Type":"ContainerStarted","Data":"1867c360e83cbc41afd656cc583c3a1cc85a0612f2dc4e35c04cac08820d6888"}
Sep 17 19:13:45.296706 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:45.296658    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:50.028661 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:50.028620    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:50.594529 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:50.594494    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" event={"ID":"74b800d0-0a51-4383-ac5a-91863ce015d6","Type":"ContainerStarted","Data":"74f9df2c8c2d7e20b9d3d7d9e306b3f75723062c750e9c48249eb07fce24187b"}
Sep 17 19:13:51.131045 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:51.131010    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:51.213733 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:51.213618    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:51.600359 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:51.600323    2587 generic.go:356] "Generic (PLEG): container finished" podID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerID="74f9df2c8c2d7e20b9d3d7d9e306b3f75723062c750e9c48249eb07fce24187b" exitCode=0
Sep 17 19:13:51.600533 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:51.600410    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" event={"ID":"74b800d0-0a51-4383-ac5a-91863ce015d6","Type":"ContainerDied","Data":"74f9df2c8c2d7e20b9d3d7d9e306b3f75723062c750e9c48249eb07fce24187b"}
Sep 17 19:13:52.608397 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:52.608312    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" event={"ID":"74b800d0-0a51-4383-ac5a-91863ce015d6","Type":"ContainerStarted","Data":"36d5629f950f1146a736c6420810a3833a6567a7c772231074010c1e1f93448e"}
Sep 17 19:13:52.608397 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:52.608388    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" event={"ID":"74b800d0-0a51-4383-ac5a-91863ce015d6","Type":"ContainerStarted","Data":"7b67ae6c3892a1407a6f835ce017a244a3ae6dac788cee5c9de5c794027d7cec"}
Sep 17 19:13:52.618365 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:52.618334    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:13:52.651319 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:52.651252    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podStartSLOduration=2.729550785 podStartE2EDuration="11.651229683s" totalImagesPullingTime="7.735809019s" totalInitContainerRuntime="1.185869879s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:52.648966313 +0000 UTC m=+2084.527935065" watchObservedRunningTime="2026-09-17 19:13:52.651229683 +0000 UTC m=+2084.530198442"
Sep 17 19:13:59.796888 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:59.796850    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:59.893821 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:13:59.893781    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:14:01.618658 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:01.618616    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:14:01.620345 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:01.620314    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 19:14:10.310048 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:10.310017    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:14:10.313045 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:10.313014    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:14:11.618753 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:11.618708    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 19:14:21.618421 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:21.618369    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 19:14:31.618453 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:31.618407    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 19:14:41.618704 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:41.618659    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 19:14:49.242908 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.242866    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:14:49.256371 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.256339    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"]
Sep 17 19:14:49.266730 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:14:49.266691    2587 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:14:49.266730 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:14:49.266730 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:14:49.266730 ip-10-0-140-194 kubenswrapper[2587]:  > containerID="962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8" cmd=["/bin/sleep","15"]
Sep 17 19:14:49.267143 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.266744    2587 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s" message="" err=<
Sep 17 19:14:49.267143 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:14:49.267143 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:14:49.267143 ip-10-0-140-194 kubenswrapper[2587]:  >
Sep 17 19:14:49.267143 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:14:49.266756    2587 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:14:49.267143 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:14:49.267143 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:14:49.267143 ip-10-0-140-194 kubenswrapper[2587]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s" podUID="e1180168-c501-4286-b7ed-14b8e394713c" containerName="main" containerID="cri-o://962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8"
Sep 17 19:14:49.267143 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.266809    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s" podUID="e1180168-c501-4286-b7ed-14b8e394713c" containerName="main" containerID="cri-o://962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8" gracePeriod=60
Sep 17 19:14:49.271750 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.271713    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s" podUID="e1180168-c501-4286-b7ed-14b8e394713c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 17 19:14:49.335474 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.335417    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:14:49.624112 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.624085    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:14:49.772237 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.772143    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-dshm\") pod \"e1180168-c501-4286-b7ed-14b8e394713c\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") "
Sep 17 19:14:49.772237 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.772209    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-model-cache\") pod \"e1180168-c501-4286-b7ed-14b8e394713c\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") "
Sep 17 19:14:49.772482 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.772282    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1180168-c501-4286-b7ed-14b8e394713c-tls-certs\") pod \"e1180168-c501-4286-b7ed-14b8e394713c\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") "
Sep 17 19:14:49.772482 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.772321    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-home\") pod \"e1180168-c501-4286-b7ed-14b8e394713c\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") "
Sep 17 19:14:49.772482 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.772374    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s65f7\" (UniqueName: \"kubernetes.io/projected/e1180168-c501-4286-b7ed-14b8e394713c-kube-api-access-s65f7\") pod \"e1180168-c501-4286-b7ed-14b8e394713c\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") "
Sep 17 19:14:49.772641 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.772484    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1180168-c501-4286-b7ed-14b8e394713c" (UID: "e1180168-c501-4286-b7ed-14b8e394713c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:14:49.772641 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.772509    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-tmp-dir\") pod \"e1180168-c501-4286-b7ed-14b8e394713c\" (UID: \"e1180168-c501-4286-b7ed-14b8e394713c\") "
Sep 17 19:14:49.772765 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.772633    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-home" (OuterVolumeSpecName: "home") pod "e1180168-c501-4286-b7ed-14b8e394713c" (UID: "e1180168-c501-4286-b7ed-14b8e394713c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:14:49.772819 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.772778    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1180168-c501-4286-b7ed-14b8e394713c" (UID: "e1180168-c501-4286-b7ed-14b8e394713c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:14:49.772879 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.772862    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:14:49.772934 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.772881    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:14:49.772934 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.772894    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:14:49.774721 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.774691    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1180168-c501-4286-b7ed-14b8e394713c-kube-api-access-s65f7" (OuterVolumeSpecName: "kube-api-access-s65f7") pod "e1180168-c501-4286-b7ed-14b8e394713c" (UID: "e1180168-c501-4286-b7ed-14b8e394713c"). InnerVolumeSpecName "kube-api-access-s65f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:14:49.774721 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.774694    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-dshm" (OuterVolumeSpecName: "dshm") pod "e1180168-c501-4286-b7ed-14b8e394713c" (UID: "e1180168-c501-4286-b7ed-14b8e394713c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:14:49.774879 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.774767    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1180168-c501-4286-b7ed-14b8e394713c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1180168-c501-4286-b7ed-14b8e394713c" (UID: "e1180168-c501-4286-b7ed-14b8e394713c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:14:49.873606 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.873513    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1180168-c501-4286-b7ed-14b8e394713c-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:14:49.873606 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.873549    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s65f7\" (UniqueName: \"kubernetes.io/projected/e1180168-c501-4286-b7ed-14b8e394713c-kube-api-access-s65f7\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:14:49.873606 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.873563    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1180168-c501-4286-b7ed-14b8e394713c-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:14:49.881731 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.881697    2587 generic.go:356] "Generic (PLEG): container finished" podID="e1180168-c501-4286-b7ed-14b8e394713c" containerID="962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8" exitCode=0
Sep 17 19:14:49.881900 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.881742    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s" event={"ID":"e1180168-c501-4286-b7ed-14b8e394713c","Type":"ContainerDied","Data":"962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8"}
Sep 17 19:14:49.881900 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.881764    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"
Sep 17 19:14:49.881900 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.881793    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s" event={"ID":"e1180168-c501-4286-b7ed-14b8e394713c","Type":"ContainerDied","Data":"bbf34cfc072130dbaf031b54b530e05f708cc8eae801f944207a9e79ba2d40a5"}
Sep 17 19:14:49.881900 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.881822    2587 scope.go:117] "RemoveContainer" containerID="962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8"
Sep 17 19:14:49.890711 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.890686    2587 scope.go:117] "RemoveContainer" containerID="962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8"
Sep 17 19:14:49.891021 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:14:49.890999    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8\": container with ID starting with 962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8 not found: ID does not exist" containerID="962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8"
Sep 17 19:14:49.891113 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.891029    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8"} err="failed to get container status \"962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8\": rpc error: code = NotFound desc = could not find container \"962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8\": container with ID starting with 962db032ed77c99d014d44c08795d6ee5301d7db408c8d22931b314e9e2ef8e8 not found: ID does not exist"
Sep 17 19:14:49.905856 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.905830    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"]
Sep 17 19:14:49.911954 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:49.911906    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969lpj5s"]
Sep 17 19:14:50.258491 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:50.258450    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1180168-c501-4286-b7ed-14b8e394713c" path="/var/lib/kubelet/pods/e1180168-c501-4286-b7ed-14b8e394713c/volumes"
Sep 17 19:14:51.618538 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:14:51.618495    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 19:15:01.618181 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:01.618131    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 19:15:11.618285 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:11.618237    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 19:15:14.942729 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:14.942686    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:21.618456 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:21.618402    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 19:15:31.619003 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:31.618935    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 19:15:39.315266 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:39.315228    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:41.149376 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:41.149339    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:41.618800 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:41.618738    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 19:15:45.682987 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:45.682914    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:51.375958 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:51.375903    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:51.449113 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:51.449072    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:51.555550 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:51.555516    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:51.628437 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:51.628353    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:15:51.636300 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:51.636265    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:15:51.819927 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:15:51.819896    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:01.795401 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:01.795371    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:08.674417 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:08.674382    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"]
Sep 17 19:16:08.674858 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:08.674665    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="modelcar" containerID="cri-o://36d5629f950f1146a736c6420810a3833a6567a7c772231074010c1e1f93448e" gracePeriod=60
Sep 17 19:16:08.683587 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:08.683555    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:08.778504 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:08.778467    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:09.250600 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:09.250571    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8_74b800d0-0a51-4383-ac5a-91863ce015d6/modelcar/0.log"
Sep 17 19:16:09.253846 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:09.253818    2587 generic.go:356] "Generic (PLEG): container finished" podID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerID="36d5629f950f1146a736c6420810a3833a6567a7c772231074010c1e1f93448e" exitCode=143
Sep 17 19:16:09.254020 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:09.253899    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" event={"ID":"74b800d0-0a51-4383-ac5a-91863ce015d6","Type":"ContainerDied","Data":"36d5629f950f1146a736c6420810a3833a6567a7c772231074010c1e1f93448e"}
Sep 17 19:16:23.698776 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:23.698675    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main" containerID="cri-o://7b67ae6c3892a1407a6f835ce017a244a3ae6dac788cee5c9de5c794027d7cec" gracePeriod=45
Sep 17 19:16:24.325589 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:24.325559    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8_74b800d0-0a51-4383-ac5a-91863ce015d6/modelcar/0.log"
Sep 17 19:16:24.326328 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:24.326303    2587 generic.go:356] "Generic (PLEG): container finished" podID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerID="7b67ae6c3892a1407a6f835ce017a244a3ae6dac788cee5c9de5c794027d7cec" exitCode=0
Sep 17 19:16:24.326431 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:24.326371    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" event={"ID":"74b800d0-0a51-4383-ac5a-91863ce015d6","Type":"ContainerDied","Data":"7b67ae6c3892a1407a6f835ce017a244a3ae6dac788cee5c9de5c794027d7cec"}
Sep 17 19:16:24.628988 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:24.628915    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 17 19:16:24.960121 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:24.960098    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8_74b800d0-0a51-4383-ac5a-91863ce015d6/modelcar/0.log"
Sep 17 19:16:24.962908 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:24.962886    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:16:25.096791 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.096753    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-tmp-dir\") pod \"74b800d0-0a51-4383-ac5a-91863ce015d6\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") "
Sep 17 19:16:25.096791 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.096797    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-dshm\") pod \"74b800d0-0a51-4383-ac5a-91863ce015d6\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") "
Sep 17 19:16:25.097070 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.096819    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-home\") pod \"74b800d0-0a51-4383-ac5a-91863ce015d6\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") "
Sep 17 19:16:25.097070 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.096850    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74b800d0-0a51-4383-ac5a-91863ce015d6-tls-certs\") pod \"74b800d0-0a51-4383-ac5a-91863ce015d6\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") "
Sep 17 19:16:25.097070 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.096921    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpwqs\" (UniqueName: \"kubernetes.io/projected/74b800d0-0a51-4383-ac5a-91863ce015d6-kube-api-access-jpwqs\") pod \"74b800d0-0a51-4383-ac5a-91863ce015d6\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") "
Sep 17 19:16:25.097070 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.096984    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-kserve-provision-location\") pod \"74b800d0-0a51-4383-ac5a-91863ce015d6\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") "
Sep 17 19:16:25.097070 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.097021    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-model-cache\") pod \"74b800d0-0a51-4383-ac5a-91863ce015d6\" (UID: \"74b800d0-0a51-4383-ac5a-91863ce015d6\") "
Sep 17 19:16:25.097435 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.097383    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "74b800d0-0a51-4383-ac5a-91863ce015d6" (UID: "74b800d0-0a51-4383-ac5a-91863ce015d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:25.097629 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.097601    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-home" (OuterVolumeSpecName: "home") pod "74b800d0-0a51-4383-ac5a-91863ce015d6" (UID: "74b800d0-0a51-4383-ac5a-91863ce015d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:25.097754 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.097664    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74b800d0-0a51-4383-ac5a-91863ce015d6" (UID: "74b800d0-0a51-4383-ac5a-91863ce015d6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:25.099434 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.099413    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-dshm" (OuterVolumeSpecName: "dshm") pod "74b800d0-0a51-4383-ac5a-91863ce015d6" (UID: "74b800d0-0a51-4383-ac5a-91863ce015d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:25.099522 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.099463    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b800d0-0a51-4383-ac5a-91863ce015d6-kube-api-access-jpwqs" (OuterVolumeSpecName: "kube-api-access-jpwqs") pod "74b800d0-0a51-4383-ac5a-91863ce015d6" (UID: "74b800d0-0a51-4383-ac5a-91863ce015d6"). InnerVolumeSpecName "kube-api-access-jpwqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:16:25.099565 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.099544    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b800d0-0a51-4383-ac5a-91863ce015d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74b800d0-0a51-4383-ac5a-91863ce015d6" (UID: "74b800d0-0a51-4383-ac5a-91863ce015d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:16:25.108673 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.108648    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "74b800d0-0a51-4383-ac5a-91863ce015d6" (UID: "74b800d0-0a51-4383-ac5a-91863ce015d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:25.198058 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.197960    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpwqs\" (UniqueName: \"kubernetes.io/projected/74b800d0-0a51-4383-ac5a-91863ce015d6-kube-api-access-jpwqs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:16:25.198058 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.198002    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:16:25.198058 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.198015    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:16:25.198058 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.198023    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:16:25.198058 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.198030    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:16:25.198058 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.198037    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74b800d0-0a51-4383-ac5a-91863ce015d6-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:16:25.198058 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.198044    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74b800d0-0a51-4383-ac5a-91863ce015d6-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:16:25.332237 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.332212    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8_74b800d0-0a51-4383-ac5a-91863ce015d6/modelcar/0.log"
Sep 17 19:16:25.333001 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.332977    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8" event={"ID":"74b800d0-0a51-4383-ac5a-91863ce015d6","Type":"ContainerDied","Data":"1867c360e83cbc41afd656cc583c3a1cc85a0612f2dc4e35c04cac08820d6888"}
Sep 17 19:16:25.333117 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.333027    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"
Sep 17 19:16:25.333117 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.333043    2587 scope.go:117] "RemoveContainer" containerID="36d5629f950f1146a736c6420810a3833a6567a7c772231074010c1e1f93448e"
Sep 17 19:16:25.342338 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.342323    2587 scope.go:117] "RemoveContainer" containerID="7b67ae6c3892a1407a6f835ce017a244a3ae6dac788cee5c9de5c794027d7cec"
Sep 17 19:16:25.350220 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.350199    2587 scope.go:117] "RemoveContainer" containerID="74f9df2c8c2d7e20b9d3d7d9e306b3f75723062c750e9c48249eb07fce24187b"
Sep 17 19:16:25.354300 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.354273    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"]
Sep 17 19:16:25.359083 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:25.359061    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dd9td8"]
Sep 17 19:16:26.258091 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:26.258050    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" path="/var/lib/kubelet/pods/74b800d0-0a51-4383-ac5a-91863ce015d6/volumes"
Sep 17 19:16:33.191006 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:33.190972    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:58.043663 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:58.043628    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:58.823751 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:58.823717    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:59.011673 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:16:59.011636    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:04.522863 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:04.522822    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:10.081677 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:10.081637    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:10.153010 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:10.152975    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:11.515265 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.515235    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"]
Sep 17 19:17:11.515674 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.515522    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="modelcar"
Sep 17 19:17:11.515674 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.515533    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="modelcar"
Sep 17 19:17:11.515674 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.515548    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main"
Sep 17 19:17:11.515674 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.515555    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main"
Sep 17 19:17:11.515674 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.515564    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1180168-c501-4286-b7ed-14b8e394713c" containerName="main"
Sep 17 19:17:11.515674 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.515571    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1180168-c501-4286-b7ed-14b8e394713c" containerName="main"
Sep 17 19:17:11.515674 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.515581    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="modelcar-init"
Sep 17 19:17:11.515674 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.515587    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="modelcar-init"
Sep 17 19:17:11.515674 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.515647    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="main"
Sep 17 19:17:11.515674 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.515656    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1180168-c501-4286-b7ed-14b8e394713c" containerName="main"
Sep 17 19:17:11.515674 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.515662    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b800d0-0a51-4383-ac5a-91863ce015d6" containerName="modelcar"
Sep 17 19:17:11.519895 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.519872    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.524677 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.524637    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 19:17:11.524677 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.524636    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 19:17:11.524971 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.524823    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:17:11.524971 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.524877    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 17 19:17:11.525120 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.525100    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-rj94h\""
Sep 17 19:17:11.525429 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.525398    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 19:17:11.541863 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.541834    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"]
Sep 17 19:17:11.574098 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.574071    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-model-cache\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.574308 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.574108    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-tmp-dir\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.574308 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.574126    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.574308 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.574152    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-dshm\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.574308 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.574265    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-cabundle-cert\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.574308 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.574303    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctf2n\" (UniqueName: \"kubernetes.io/projected/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-kube-api-access-ctf2n\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.574537 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.574360    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-home\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.574537 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.574392    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-tls-certs\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.588558 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.588527    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:11.675426 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.675388    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-model-cache\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.675426 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.675430    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-tmp-dir\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.675675 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.675458    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.675675 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.675492    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-dshm\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.675675 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.675525    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-cabundle-cert\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.675675 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.675551    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctf2n\" (UniqueName: \"kubernetes.io/projected/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-kube-api-access-ctf2n\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.675675 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.675593    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-home\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.675675 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.675618    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-tls-certs\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.675982 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.675844    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-tmp-dir\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.675982 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.675903    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-model-cache\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.676101 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.676007    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-home\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.676101 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.676020    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.676272 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.676251    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-cabundle-cert\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.678195 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.678175    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-dshm\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.678336 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.678317    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-tls-certs\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.687697 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.687664    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctf2n\" (UniqueName: \"kubernetes.io/projected/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-kube-api-access-ctf2n\") pod \"custom-route-timeout-test-kserve-6dd4456bf-5f9ph\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.830922 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.830821    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:11.878316 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:11.878280    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:12.118314 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:12.118284    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"]
Sep 17 19:17:12.119921 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:17:12.119895    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017bf528_e8f9_4ad0_a70a_3b9e34958c3c.slice/crio-7f7f24e7048e2d9d6515027d1dd5046428fadd73d3711e7024db8640d8f62682 WatchSource:0}: Error finding container 7f7f24e7048e2d9d6515027d1dd5046428fadd73d3711e7024db8640d8f62682: Status 404 returned error can't find the container with id 7f7f24e7048e2d9d6515027d1dd5046428fadd73d3711e7024db8640d8f62682
Sep 17 19:17:12.548584 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:12.548548    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" event={"ID":"017bf528-e8f9-4ad0-a70a-3b9e34958c3c","Type":"ContainerStarted","Data":"05b7717c011d9579b8f8c81312a1bc8d849de543925a11fb1765944993b1f525"}
Sep 17 19:17:12.549053 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:12.548602    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" event={"ID":"017bf528-e8f9-4ad0-a70a-3b9e34958c3c","Type":"ContainerStarted","Data":"7f7f24e7048e2d9d6515027d1dd5046428fadd73d3711e7024db8640d8f62682"}
Sep 17 19:17:14.558259 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:14.558222    2587 generic.go:356] "Generic (PLEG): container finished" podID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerID="05b7717c011d9579b8f8c81312a1bc8d849de543925a11fb1765944993b1f525" exitCode=0
Sep 17 19:17:14.558671 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:14.558294    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" event={"ID":"017bf528-e8f9-4ad0-a70a-3b9e34958c3c","Type":"ContainerDied","Data":"05b7717c011d9579b8f8c81312a1bc8d849de543925a11fb1765944993b1f525"}
Sep 17 19:17:15.565495 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:15.565461    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" event={"ID":"017bf528-e8f9-4ad0-a70a-3b9e34958c3c","Type":"ContainerStarted","Data":"9c13fb127331a2a37dea6b90c73dd57cb9e2e14ea7cac6564cc96484b71eb12d"}
Sep 17 19:17:15.606136 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:15.606074    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" podStartSLOduration=2.38636246 podStartE2EDuration="4.606054385s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.219691925s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:17:15.605734241 +0000 UTC m=+2287.484702999" watchObservedRunningTime="2026-09-17 19:17:15.606054385 +0000 UTC m=+2287.485023143"
Sep 17 19:17:15.831815 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:15.831723    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:18.924629 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:18.924588    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:21.831147 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:21.831097    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:17:21.832899 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:21.832868    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 19:17:31.831536 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:31.831492    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 19:17:41.831367 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:41.831314    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 19:17:47.744748 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:47.744707    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:47.879394 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:47.879357    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:51.831686 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:17:51.831592    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 19:18:01.831957 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:18:01.831903    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 19:18:11.832005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:18:11.831923    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 19:18:21.831336 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:18:21.831280    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 19:18:31.831512 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:18:31.831471    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 19:18:37.690647 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:18:37.690598    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:37.811377 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:18:37.811326    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:41.831791 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:18:41.831737    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 19:18:44.292581 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:18:44.292549    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:49.978347 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:18:49.978303    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:50.051584 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:18:50.051544    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:51.831320 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:18:51.831279    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 19:18:58.197621 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:18:58.197575    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:58.513276 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:18:58.513237    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:01.831419 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:01.831376    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 19:19:05.177628 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:05.177589    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:10.336872 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:10.336842    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:19:10.341143 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:10.341122    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:19:11.842283 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:11.842251    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:19:11.856085 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:11.851032    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:19:16.717756 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:16.717711    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:16.718860 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:16.718833    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"]
Sep 17 19:19:16.793911 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:16.793876    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:31.742228 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:31.742120    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main" containerID="cri-o://9c13fb127331a2a37dea6b90c73dd57cb9e2e14ea7cac6564cc96484b71eb12d" gracePeriod=45
Sep 17 19:19:31.843073 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:31.843032    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 19:19:32.495478 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.495449    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:19:32.678015 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.677900    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctf2n\" (UniqueName: \"kubernetes.io/projected/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-kube-api-access-ctf2n\") pod \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") "
Sep 17 19:19:32.678015 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.677988    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-model-cache\") pod \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") "
Sep 17 19:19:32.678015 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.678006    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-home\") pod \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") "
Sep 17 19:19:32.678333 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.678032    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-tls-certs\") pod \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") "
Sep 17 19:19:32.678333 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.678076    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-kserve-provision-location\") pod \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") "
Sep 17 19:19:32.678333 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.678099    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-cabundle-cert\") pod \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") "
Sep 17 19:19:32.678333 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.678128    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-dshm\") pod \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") "
Sep 17 19:19:32.678333 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.678180    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-tmp-dir\") pod \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\" (UID: \"017bf528-e8f9-4ad0-a70a-3b9e34958c3c\") "
Sep 17 19:19:32.678333 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.678273    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-model-cache" (OuterVolumeSpecName: "model-cache") pod "017bf528-e8f9-4ad0-a70a-3b9e34958c3c" (UID: "017bf528-e8f9-4ad0-a70a-3b9e34958c3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:32.678650 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.678470    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:19:32.678650 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.678551    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "017bf528-e8f9-4ad0-a70a-3b9e34958c3c" (UID: "017bf528-e8f9-4ad0-a70a-3b9e34958c3c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:19:32.678830 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.678808    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-home" (OuterVolumeSpecName: "home") pod "017bf528-e8f9-4ad0-a70a-3b9e34958c3c" (UID: "017bf528-e8f9-4ad0-a70a-3b9e34958c3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:32.681192 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.681152    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-dshm" (OuterVolumeSpecName: "dshm") pod "017bf528-e8f9-4ad0-a70a-3b9e34958c3c" (UID: "017bf528-e8f9-4ad0-a70a-3b9e34958c3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:32.681192 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.681166    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-kube-api-access-ctf2n" (OuterVolumeSpecName: "kube-api-access-ctf2n") pod "017bf528-e8f9-4ad0-a70a-3b9e34958c3c" (UID: "017bf528-e8f9-4ad0-a70a-3b9e34958c3c"). InnerVolumeSpecName "kube-api-access-ctf2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:19:32.681334 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.681236    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "017bf528-e8f9-4ad0-a70a-3b9e34958c3c" (UID: "017bf528-e8f9-4ad0-a70a-3b9e34958c3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:19:32.690180 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.690161    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "017bf528-e8f9-4ad0-a70a-3b9e34958c3c" (UID: "017bf528-e8f9-4ad0-a70a-3b9e34958c3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:32.708525 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.708490    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "017bf528-e8f9-4ad0-a70a-3b9e34958c3c" (UID: "017bf528-e8f9-4ad0-a70a-3b9e34958c3c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:32.779813 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.779777    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctf2n\" (UniqueName: \"kubernetes.io/projected/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-kube-api-access-ctf2n\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:19:32.779813 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.779809    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:19:32.779813 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.779820    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:19:32.780290 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.779829    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:19:32.780290 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.779838    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:19:32.780290 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.779845    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:19:32.780290 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:32.779852    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017bf528-e8f9-4ad0-a70a-3b9e34958c3c-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:19:33.217481 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:33.217447    2587 generic.go:356] "Generic (PLEG): container finished" podID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerID="9c13fb127331a2a37dea6b90c73dd57cb9e2e14ea7cac6564cc96484b71eb12d" exitCode=0
Sep 17 19:19:33.217667 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:33.217501    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"
Sep 17 19:19:33.217667 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:33.217530    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" event={"ID":"017bf528-e8f9-4ad0-a70a-3b9e34958c3c","Type":"ContainerDied","Data":"9c13fb127331a2a37dea6b90c73dd57cb9e2e14ea7cac6564cc96484b71eb12d"}
Sep 17 19:19:33.217667 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:33.217590    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph" event={"ID":"017bf528-e8f9-4ad0-a70a-3b9e34958c3c","Type":"ContainerDied","Data":"7f7f24e7048e2d9d6515027d1dd5046428fadd73d3711e7024db8640d8f62682"}
Sep 17 19:19:33.217667 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:33.217621    2587 scope.go:117] "RemoveContainer" containerID="9c13fb127331a2a37dea6b90c73dd57cb9e2e14ea7cac6564cc96484b71eb12d"
Sep 17 19:19:33.227405 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:33.227388    2587 scope.go:117] "RemoveContainer" containerID="05b7717c011d9579b8f8c81312a1bc8d849de543925a11fb1765944993b1f525"
Sep 17 19:19:33.235176 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:33.235157    2587 scope.go:117] "RemoveContainer" containerID="9c13fb127331a2a37dea6b90c73dd57cb9e2e14ea7cac6564cc96484b71eb12d"
Sep 17 19:19:33.235440 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:19:33.235417    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c13fb127331a2a37dea6b90c73dd57cb9e2e14ea7cac6564cc96484b71eb12d\": container with ID starting with 9c13fb127331a2a37dea6b90c73dd57cb9e2e14ea7cac6564cc96484b71eb12d not found: ID does not exist" containerID="9c13fb127331a2a37dea6b90c73dd57cb9e2e14ea7cac6564cc96484b71eb12d"
Sep 17 19:19:33.235528 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:33.235448    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c13fb127331a2a37dea6b90c73dd57cb9e2e14ea7cac6564cc96484b71eb12d"} err="failed to get container status \"9c13fb127331a2a37dea6b90c73dd57cb9e2e14ea7cac6564cc96484b71eb12d\": rpc error: code = NotFound desc = could not find container \"9c13fb127331a2a37dea6b90c73dd57cb9e2e14ea7cac6564cc96484b71eb12d\": container with ID starting with 9c13fb127331a2a37dea6b90c73dd57cb9e2e14ea7cac6564cc96484b71eb12d not found: ID does not exist"
Sep 17 19:19:33.235577 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:33.235532    2587 scope.go:117] "RemoveContainer" containerID="05b7717c011d9579b8f8c81312a1bc8d849de543925a11fb1765944993b1f525"
Sep 17 19:19:33.235804 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:19:33.235787    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b7717c011d9579b8f8c81312a1bc8d849de543925a11fb1765944993b1f525\": container with ID starting with 05b7717c011d9579b8f8c81312a1bc8d849de543925a11fb1765944993b1f525 not found: ID does not exist" containerID="05b7717c011d9579b8f8c81312a1bc8d849de543925a11fb1765944993b1f525"
Sep 17 19:19:33.235846 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:33.235810    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b7717c011d9579b8f8c81312a1bc8d849de543925a11fb1765944993b1f525"} err="failed to get container status \"05b7717c011d9579b8f8c81312a1bc8d849de543925a11fb1765944993b1f525\": rpc error: code = NotFound desc = could not find container \"05b7717c011d9579b8f8c81312a1bc8d849de543925a11fb1765944993b1f525\": container with ID starting with 05b7717c011d9579b8f8c81312a1bc8d849de543925a11fb1765944993b1f525 not found: ID does not exist"
Sep 17 19:19:33.240059 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:33.240035    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"]
Sep 17 19:19:33.243798 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:33.243778    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-6dd4456bf-5f9ph"]
Sep 17 19:19:34.258153 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:19:34.258117    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" path="/var/lib/kubelet/pods/017bf528-e8f9-4ad0-a70a-3b9e34958c3c/volumes"
Sep 17 19:20:01.110755 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:01.110717    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:06.749054 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:06.749018    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:06.878864 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:06.878823    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:07.101231 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:07.101193    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:07.203029 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:07.202995    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:10.828042 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:10.827994    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:13.712730 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:13.712685    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:19.341117 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:19.341073    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:19.442475 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:19.442438    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:31.800659 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.800621    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"]
Sep 17 19:20:31.801167 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.801148    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="storage-initializer"
Sep 17 19:20:31.801221 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.801171    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="storage-initializer"
Sep 17 19:20:31.801221 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.801194    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main"
Sep 17 19:20:31.801221 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.801203    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main"
Sep 17 19:20:31.801329 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.801283    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="017bf528-e8f9-4ad0-a70a-3b9e34958c3c" containerName="main"
Sep 17 19:20:31.806000 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.805974    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:31.808698 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.808668    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 19:20:31.808698 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.808669    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 19:20:31.808918 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.808733    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 17 19:20:31.809116 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.809100    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:20:31.809675 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.809658    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-h4tdj\""
Sep 17 19:20:31.809720 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.809700    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 19:20:31.817207 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.817183    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"]
Sep 17 19:20:31.841504 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.841465    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:31.947364 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.947328    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-home\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:31.947364 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.947367    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-model-cache\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:31.947639 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.947427    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-tmp-dir\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:31.947639 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.947477    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-cabundle-cert\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:31.947639 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.947520    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-tls-certs\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:31.947639 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.947580    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-kserve-provision-location\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:31.947639 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.947601    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-dshm\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:31.947639 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:31.947624    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkc5h\" (UniqueName: \"kubernetes.io/projected/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-kube-api-access-kkc5h\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.048677 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.048637    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-home\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.048677 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.048675    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-model-cache\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.048985 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.048693    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-tmp-dir\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.048985 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.048823    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-cabundle-cert\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.048985 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.048882    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-tls-certs\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.048985 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.048925    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-kserve-provision-location\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.048985 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.048984    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-dshm\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.049258 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.049023    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkc5h\" (UniqueName: \"kubernetes.io/projected/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-kube-api-access-kkc5h\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.049258 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.049117    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-model-cache\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.049258 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.049154    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-tmp-dir\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.049258 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.049210    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-home\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.049495 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.049416    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-kserve-provision-location\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.049495 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.049487    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-cabundle-cert\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.051577 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.051519    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-dshm\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.051850 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.051826    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-tls-certs\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.057475 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.057448    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkc5h\" (UniqueName: \"kubernetes.io/projected/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-kube-api-access-kkc5h\") pod \"router-with-refs-test-kserve-7f85fdbc65-rxzzb\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.117700 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.117650    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:32.193051 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.193001    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:32.289788 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.289752    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:32.403110 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.403080    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"]
Sep 17 19:20:32.405048 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:20:32.405015    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59d70bf_cfc7_4d52_94d0_c664e5a74e53.slice/crio-2bb4be7becca8e7185667c24e6074cd778fcedc49bf87739c2502c78234f1ad8 WatchSource:0}: Error finding container 2bb4be7becca8e7185667c24e6074cd778fcedc49bf87739c2502c78234f1ad8: Status 404 returned error can't find the container with id 2bb4be7becca8e7185667c24e6074cd778fcedc49bf87739c2502c78234f1ad8
Sep 17 19:20:32.466760 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:32.466732    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" event={"ID":"f59d70bf-cfc7-4d52-94d0-c664e5a74e53","Type":"ContainerStarted","Data":"2bb4be7becca8e7185667c24e6074cd778fcedc49bf87739c2502c78234f1ad8"}
Sep 17 19:20:33.473056 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:33.473018    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" event={"ID":"f59d70bf-cfc7-4d52-94d0-c664e5a74e53","Type":"ContainerStarted","Data":"10e06d153a2dd5fe81545d0994261249924c6f8276b85bce345f76ab6530558a"}
Sep 17 19:20:35.483071 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:35.483038    2587 generic.go:356] "Generic (PLEG): container finished" podID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerID="10e06d153a2dd5fe81545d0994261249924c6f8276b85bce345f76ab6530558a" exitCode=0
Sep 17 19:20:35.483528 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:35.483116    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" event={"ID":"f59d70bf-cfc7-4d52-94d0-c664e5a74e53","Type":"ContainerDied","Data":"10e06d153a2dd5fe81545d0994261249924c6f8276b85bce345f76ab6530558a"}
Sep 17 19:20:36.490035 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:36.489995    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" event={"ID":"f59d70bf-cfc7-4d52-94d0-c664e5a74e53","Type":"ContainerStarted","Data":"5e2d15167e381ecf2cd2de39bc175584dbc696026299721a5c607772ddc57dcc"}
Sep 17 19:20:36.514052 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:36.513981    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" podStartSLOduration=3.290134694 podStartE2EDuration="5.513931582s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.223796888s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:20:36.512550763 +0000 UTC m=+2488.391519520" watchObservedRunningTime="2026-09-17 19:20:36.513931582 +0000 UTC m=+2488.392900340"
Sep 17 19:20:37.118023 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:37.117986    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:42.118095 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:42.118048    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:20:42.119618 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:42.119588    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 19:20:46.249824 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:46.249782    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:47.507877 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:47.507837    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:52.118630 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:20:52.118522    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 19:21:02.118401 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:02.118346    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 19:21:12.118525 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:12.118473    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 19:21:22.118771 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:22.118722    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 19:21:32.119140 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:32.119089    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 19:21:36.160153 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:36.160111    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:36.250442 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:36.250399    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:42.118603 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:42.118553    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 19:21:42.530638 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:42.530593    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:43.844749 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:43.844713    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:44.117610 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:44.117508    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:48.287335 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:48.287293    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:48.388357 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:48.388313    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:51.024408 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:51.024370    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:52.118215 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:52.118161    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 19:21:56.692327 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:56.692288    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:57.231244 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:21:57.231201    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:02.119004 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:02.118923    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 19:22:12.118299 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:12.118253    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 19:22:21.681805 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:21.681713    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:22.118134 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:22.118092    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 19:22:32.128024 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:32.127992    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:22:32.135759 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:32.135730    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:22:36.997051 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:36.997016    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"]
Sep 17 19:22:36.999064 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:36.999027    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:37.082497 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:37.082464    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:52.020766 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.020724    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main" containerID="cri-o://5e2d15167e381ecf2cd2de39bc175584dbc696026299721a5c607772ddc57dcc" gracePeriod=45
Sep 17 19:22:52.128829 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.128782    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 19:22:52.772377 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.772347    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:22:52.836196 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.836105    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-model-cache\") pod \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") "
Sep 17 19:22:52.836196 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.836153    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-dshm\") pod \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") "
Sep 17 19:22:52.836196 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.836182    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-cabundle-cert\") pod \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") "
Sep 17 19:22:52.836488 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.836205    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-tls-certs\") pod \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") "
Sep 17 19:22:52.836488 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.836236    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-kserve-provision-location\") pod \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") "
Sep 17 19:22:52.836488 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.836261    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkc5h\" (UniqueName: \"kubernetes.io/projected/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-kube-api-access-kkc5h\") pod \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") "
Sep 17 19:22:52.836488 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.836306    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-tmp-dir\") pod \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") "
Sep 17 19:22:52.836488 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.836342    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-home\") pod \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\" (UID: \"f59d70bf-cfc7-4d52-94d0-c664e5a74e53\") "
Sep 17 19:22:52.836488 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.836424    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-model-cache" (OuterVolumeSpecName: "model-cache") pod "f59d70bf-cfc7-4d52-94d0-c664e5a74e53" (UID: "f59d70bf-cfc7-4d52-94d0-c664e5a74e53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:52.836784 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.836577    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:22:52.836784 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.836612    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f59d70bf-cfc7-4d52-94d0-c664e5a74e53" (UID: "f59d70bf-cfc7-4d52-94d0-c664e5a74e53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:22:52.837328 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.837295    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-home" (OuterVolumeSpecName: "home") pod "f59d70bf-cfc7-4d52-94d0-c664e5a74e53" (UID: "f59d70bf-cfc7-4d52-94d0-c664e5a74e53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:52.838769 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.838742    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f59d70bf-cfc7-4d52-94d0-c664e5a74e53" (UID: "f59d70bf-cfc7-4d52-94d0-c664e5a74e53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:22:52.838994 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.838970    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-dshm" (OuterVolumeSpecName: "dshm") pod "f59d70bf-cfc7-4d52-94d0-c664e5a74e53" (UID: "f59d70bf-cfc7-4d52-94d0-c664e5a74e53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:52.839094 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.839016    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-kube-api-access-kkc5h" (OuterVolumeSpecName: "kube-api-access-kkc5h") pod "f59d70bf-cfc7-4d52-94d0-c664e5a74e53" (UID: "f59d70bf-cfc7-4d52-94d0-c664e5a74e53"). InnerVolumeSpecName "kube-api-access-kkc5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:22:52.848253 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.848222    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f59d70bf-cfc7-4d52-94d0-c664e5a74e53" (UID: "f59d70bf-cfc7-4d52-94d0-c664e5a74e53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:52.860058 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.860024    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f59d70bf-cfc7-4d52-94d0-c664e5a74e53" (UID: "f59d70bf-cfc7-4d52-94d0-c664e5a74e53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:52.937440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.937400    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:22:52.937440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.937440    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:22:52.937616 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.937454    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:22:52.937616 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.937464    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:22:52.937616 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.937473    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkc5h\" (UniqueName: \"kubernetes.io/projected/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-kube-api-access-kkc5h\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:22:52.937616 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.937482    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:22:52.937616 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:52.937489    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f59d70bf-cfc7-4d52-94d0-c664e5a74e53-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:22:53.129539 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:53.129445    2587 generic.go:356] "Generic (PLEG): container finished" podID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerID="5e2d15167e381ecf2cd2de39bc175584dbc696026299721a5c607772ddc57dcc" exitCode=0
Sep 17 19:22:53.129539 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:53.129501    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"
Sep 17 19:22:53.129539 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:53.129518    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" event={"ID":"f59d70bf-cfc7-4d52-94d0-c664e5a74e53","Type":"ContainerDied","Data":"5e2d15167e381ecf2cd2de39bc175584dbc696026299721a5c607772ddc57dcc"}
Sep 17 19:22:53.130068 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:53.129561    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb" event={"ID":"f59d70bf-cfc7-4d52-94d0-c664e5a74e53","Type":"ContainerDied","Data":"2bb4be7becca8e7185667c24e6074cd778fcedc49bf87739c2502c78234f1ad8"}
Sep 17 19:22:53.130068 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:53.129585    2587 scope.go:117] "RemoveContainer" containerID="5e2d15167e381ecf2cd2de39bc175584dbc696026299721a5c607772ddc57dcc"
Sep 17 19:22:53.138866 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:53.138846    2587 scope.go:117] "RemoveContainer" containerID="10e06d153a2dd5fe81545d0994261249924c6f8276b85bce345f76ab6530558a"
Sep 17 19:22:53.152466 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:53.152410    2587 scope.go:117] "RemoveContainer" containerID="5e2d15167e381ecf2cd2de39bc175584dbc696026299721a5c607772ddc57dcc"
Sep 17 19:22:53.152969 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:22:53.152914    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2d15167e381ecf2cd2de39bc175584dbc696026299721a5c607772ddc57dcc\": container with ID starting with 5e2d15167e381ecf2cd2de39bc175584dbc696026299721a5c607772ddc57dcc not found: ID does not exist" containerID="5e2d15167e381ecf2cd2de39bc175584dbc696026299721a5c607772ddc57dcc"
Sep 17 19:22:53.153103 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:53.153075    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2d15167e381ecf2cd2de39bc175584dbc696026299721a5c607772ddc57dcc"} err="failed to get container status \"5e2d15167e381ecf2cd2de39bc175584dbc696026299721a5c607772ddc57dcc\": rpc error: code = NotFound desc = could not find container \"5e2d15167e381ecf2cd2de39bc175584dbc696026299721a5c607772ddc57dcc\": container with ID starting with 5e2d15167e381ecf2cd2de39bc175584dbc696026299721a5c607772ddc57dcc not found: ID does not exist"
Sep 17 19:22:53.153157 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:53.153108    2587 scope.go:117] "RemoveContainer" containerID="10e06d153a2dd5fe81545d0994261249924c6f8276b85bce345f76ab6530558a"
Sep 17 19:22:53.153443 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:22:53.153419    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e06d153a2dd5fe81545d0994261249924c6f8276b85bce345f76ab6530558a\": container with ID starting with 10e06d153a2dd5fe81545d0994261249924c6f8276b85bce345f76ab6530558a not found: ID does not exist" containerID="10e06d153a2dd5fe81545d0994261249924c6f8276b85bce345f76ab6530558a"
Sep 17 19:22:53.153505 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:53.153449    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e06d153a2dd5fe81545d0994261249924c6f8276b85bce345f76ab6530558a"} err="failed to get container status \"10e06d153a2dd5fe81545d0994261249924c6f8276b85bce345f76ab6530558a\": rpc error: code = NotFound desc = could not find container \"10e06d153a2dd5fe81545d0994261249924c6f8276b85bce345f76ab6530558a\": container with ID starting with 10e06d153a2dd5fe81545d0994261249924c6f8276b85bce345f76ab6530558a not found: ID does not exist"
Sep 17 19:22:53.153581 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:53.153564    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"]
Sep 17 19:22:53.157974 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:53.157931    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7f85fdbc65-rxzzb"]
Sep 17 19:22:54.258266 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:54.258231    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" path="/var/lib/kubelet/pods/f59d70bf-cfc7-4d52-94d0-c664e5a74e53/volumes"
Sep 17 19:22:57.867446 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:57.867415    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:57.931869 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:22:57.931827    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:27.054859 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:27.054795    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:27.162354 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:27.162310    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:32.102857 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:32.102802    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:50.226985 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.226888    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"]
Sep 17 19:23:50.227381 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.227270    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main"
Sep 17 19:23:50.227381 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.227286    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main"
Sep 17 19:23:50.227381 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.227310    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="storage-initializer"
Sep 17 19:23:50.227381 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.227315    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="storage-initializer"
Sep 17 19:23:50.227381 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.227371    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f59d70bf-cfc7-4d52-94d0-c664e5a74e53" containerName="main"
Sep 17 19:23:50.230184 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.230168    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.232995 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.232965    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 19:23:50.233246 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.233229    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 19:23:50.234155 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.234129    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 19:23:50.234155 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.234159    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:23:50.234155 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.234177    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-l2qb5\""
Sep 17 19:23:50.234456 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.234159    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-dw9zb\""
Sep 17 19:23:50.234517 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.234468    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 19:23:50.235129 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.235107    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"]
Sep 17 19:23:50.238447 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.238428    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.251175 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.251138    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"]
Sep 17 19:23:50.270243 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.270205    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"]
Sep 17 19:23:50.274526 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.274477    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:50.319803 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.319757    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.319803 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.319799    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.320060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.319853    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.320060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.319891    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hssql\" (UniqueName: \"kubernetes.io/projected/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-kube-api-access-hssql\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.320060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.319924    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.320060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.319982    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.320060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.320005    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.320060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.320040    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96f33c73-db06-4ed5-87c4-8e5b7def9a49-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.320060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.320060    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.320298 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.320074    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvh4b\" (UniqueName: \"kubernetes.io/projected/96f33c73-db06-4ed5-87c4-8e5b7def9a49-kube-api-access-pvh4b\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.320298 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.320129    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.320298 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.320155    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.320298 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.320179    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.320298 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.320206    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.320298 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.320220    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.320298 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.320240    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96f33c73-db06-4ed5-87c4-8e5b7def9a49-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.420743 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.420704    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.420743 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.420746    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.421025 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.420765    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.421025 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.420789    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hssql\" (UniqueName: \"kubernetes.io/projected/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-kube-api-access-hssql\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.421025 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.420847    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.421025 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.420888    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.421025 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.420917    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.421025 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.420956    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96f33c73-db06-4ed5-87c4-8e5b7def9a49-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.421025 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.420992    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.421025 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.421015    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvh4b\" (UniqueName: \"kubernetes.io/projected/96f33c73-db06-4ed5-87c4-8e5b7def9a49-kube-api-access-pvh4b\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.421437 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.421071    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.421437 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.421100    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.421437 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.421129    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.421437 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.421161    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.421437 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.421185    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.421437 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.421227    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.421437 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.421272    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96f33c73-db06-4ed5-87c4-8e5b7def9a49-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.421437 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.421302    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.421437 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.421343    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.421915 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.421512    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.421915 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.421739    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.421915 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.421805    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96f33c73-db06-4ed5-87c4-8e5b7def9a49-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.422119 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.422002    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.422119 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.422038    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.422119 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.422063    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.422279 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.422168    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.423985 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.423933    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.424100 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.424010    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.424408 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.424387    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96f33c73-db06-4ed5-87c4-8e5b7def9a49-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.424450 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.424420    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.432056 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.432028    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hssql\" (UniqueName: \"kubernetes.io/projected/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-kube-api-access-hssql\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.432367 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.432341    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvh4b\" (UniqueName: \"kubernetes.io/projected/96f33c73-db06-4ed5-87c4-8e5b7def9a49-kube-api-access-pvh4b\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.502850 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.502816    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:50.540966 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.540897    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:50.548268 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.548237    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:23:50.843620 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.843538    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"]
Sep 17 19:23:50.846710 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:50.846664    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"]
Sep 17 19:23:50.847046 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:23:50.847021    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f33c73_db06_4ed5_87c4_8e5b7def9a49.slice/crio-cf11a5c09eb22b3dfefc38bb882e336942220b8c40b3b777a43e5fe923cf845e WatchSource:0}: Error finding container cf11a5c09eb22b3dfefc38bb882e336942220b8c40b3b777a43e5fe923cf845e: Status 404 returned error can't find the container with id cf11a5c09eb22b3dfefc38bb882e336942220b8c40b3b777a43e5fe923cf845e
Sep 17 19:23:50.849566 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:23:50.849542    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4ac10f_5bd3_431f_a2a1_6b6efd9daea4.slice/crio-f06921b3eae8f2f35a6f8a22838a518420e054793dbcffd4091709832a4f7515 WatchSource:0}: Error finding container f06921b3eae8f2f35a6f8a22838a518420e054793dbcffd4091709832a4f7515: Status 404 returned error can't find the container with id f06921b3eae8f2f35a6f8a22838a518420e054793dbcffd4091709832a4f7515
Sep 17 19:23:51.384080 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:51.384038    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" event={"ID":"96f33c73-db06-4ed5-87c4-8e5b7def9a49","Type":"ContainerStarted","Data":"a04d459c028c5c99aa12c42dbfc0532817d799bc4e78a3373125f28ad4601177"}
Sep 17 19:23:51.384581 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:51.384139    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" event={"ID":"96f33c73-db06-4ed5-87c4-8e5b7def9a49","Type":"ContainerStarted","Data":"cf11a5c09eb22b3dfefc38bb882e336942220b8c40b3b777a43e5fe923cf845e"}
Sep 17 19:23:51.385896 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:51.385872    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" event={"ID":"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4","Type":"ContainerStarted","Data":"7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2"}
Sep 17 19:23:51.386038 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:51.385912    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" event={"ID":"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4","Type":"ContainerStarted","Data":"f06921b3eae8f2f35a6f8a22838a518420e054793dbcffd4091709832a4f7515"}
Sep 17 19:23:51.386038 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:51.385989    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:23:52.396152 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:52.396107    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" event={"ID":"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4","Type":"ContainerStarted","Data":"517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71"}
Sep 17 19:23:54.131008 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:54.130964    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:54.407634 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:54.407601    2587 generic.go:356] "Generic (PLEG): container finished" podID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerID="a04d459c028c5c99aa12c42dbfc0532817d799bc4e78a3373125f28ad4601177" exitCode=0
Sep 17 19:23:54.407834 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:54.407680    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" event={"ID":"96f33c73-db06-4ed5-87c4-8e5b7def9a49","Type":"ContainerDied","Data":"a04d459c028c5c99aa12c42dbfc0532817d799bc4e78a3373125f28ad4601177"}
Sep 17 19:23:54.409628 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:54.409606    2587 generic.go:356] "Generic (PLEG): container finished" podID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerID="517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71" exitCode=0
Sep 17 19:23:54.409734 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:54.409669    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" event={"ID":"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4","Type":"ContainerDied","Data":"517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71"}
Sep 17 19:23:55.211751 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:55.211711    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:55.418789 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:55.418743    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" event={"ID":"96f33c73-db06-4ed5-87c4-8e5b7def9a49","Type":"ContainerStarted","Data":"f484896164bbf3d6f47810370e1e0d169213e4e1781042bcf76fe5f63929a529"}
Sep 17 19:23:55.421397 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:55.421362    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" event={"ID":"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4","Type":"ContainerStarted","Data":"f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5"}
Sep 17 19:23:55.451852 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:55.451791    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podStartSLOduration=2.935907301 podStartE2EDuration="5.451772256s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.515864955s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:23:55.447149112 +0000 UTC m=+2687.326117878" watchObservedRunningTime="2026-09-17 19:23:55.451772256 +0000 UTC m=+2687.330741038"
Sep 17 19:23:55.488058 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:23:55.487989    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podStartSLOduration=3.008098936 podStartE2EDuration="5.487966096s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.47986716s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:23:55.487918391 +0000 UTC m=+2687.366887147" watchObservedRunningTime="2026-09-17 19:23:55.487966096 +0000 UTC m=+2687.366934852"
Sep 17 19:24:00.099417 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:00.099371    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:00.541788 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:00.541746    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:24:00.541997 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:00.541827    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:24:00.543365 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:00.543324    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 17 19:24:00.548440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:00.548413    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:24:00.548588 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:00.548475    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:24:00.549573 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:00.549544    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:24:00.556080 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:00.556046    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:24:01.064886 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:01.064845    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:03.576569 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:03.576523    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:03.691862 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:03.691768    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:03.769720 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:03.769678    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:10.369880 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:10.369850    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:24:10.372707 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:10.372679    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:24:10.460653 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:10.460607    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:10.541832 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:10.541783    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 17 19:24:10.548868 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:10.548821    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:24:15.798468 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:15.798424    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:15.877122 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:15.877071    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:17.828379 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:17.828332    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:17.997163 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:17.997131    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:20.542201 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:20.542132    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 17 19:24:20.548953 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:20.548905    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:24:30.541691 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:30.541635    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 17 19:24:30.548836 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:30.548794    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:24:40.541924 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:40.541864    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 17 19:24:40.548990 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:40.548929    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:24:41.490605 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:41.490567    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:50.541694 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:50.541642    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 17 19:24:50.549412 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:24:50.549353    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:25:00.541929 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:25:00.541870    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 17 19:25:00.549074 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:25:00.549030    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:25:10.541383 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:25:10.541338    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 17 19:25:10.549189 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:25:10.549154    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:25:20.542402 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:25:20.542262    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 17 19:25:20.549071 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:25:20.549028    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:25:30.542201 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:25:30.542148    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 17 19:25:30.548749 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:25:30.548717    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:25:40.542205 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:25:40.542143    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 17 19:25:40.549316 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:25:40.549280    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:25:50.541418 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:25:50.541364    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 17 19:25:50.549364 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:25:50.549322    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:26:00.541468 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:00.541405    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 17 19:26:00.548657 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:00.548615    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:26:10.541347 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:10.541296    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8001/health\": dial tcp 10.132.0.48:8001: connect: connection refused"
Sep 17 19:26:10.548870 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:10.548832    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:26:20.093825 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:20.093794    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:26:20.548862 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:20.548812    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:26:20.551933 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:20.551905    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:26:20.568519 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:20.568488    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:26:21.288396 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:21.288349    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:26:30.548918 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:30.548869    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 17 19:26:40.561357 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:40.561320    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:26:40.570093 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:40.570059    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:26:50.226471 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:50.226380    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:26:50.278808 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:50.278769    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"]
Sep 17 19:26:50.286486 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:50.286456    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"]
Sep 17 19:26:50.320270 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:26:50.320233    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:05.302854 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:05.302811    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main" containerID="cri-o://f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5" gracePeriod=45
Sep 17 19:27:05.309610 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:05.309568    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main" containerID="cri-o://f484896164bbf3d6f47810370e1e0d169213e4e1781042bcf76fe5f63929a529" gracePeriod=45
Sep 17 19:27:06.115815 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.115782    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="llm-d-routing-sidecar" containerID="cri-o://7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2" gracePeriod=45
Sep 17 19:27:06.271362 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.271329    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:27:06.360620 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.360587    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:27:06.368110 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368075    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-cabundle-cert\") pod \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") "
Sep 17 19:27:06.368309 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368123    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-model-cache\") pod \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") "
Sep 17 19:27:06.368309 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368149    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96f33c73-db06-4ed5-87c4-8e5b7def9a49-cabundle-cert\") pod \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") "
Sep 17 19:27:06.368309 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368177    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-tmp-dir\") pod \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") "
Sep 17 19:27:06.368309 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368207    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-dshm\") pod \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") "
Sep 17 19:27:06.368309 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368228    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-dshm\") pod \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") "
Sep 17 19:27:06.368309 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368270    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-tls-certs\") pod \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") "
Sep 17 19:27:06.368309 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368300    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-home\") pod \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") "
Sep 17 19:27:06.368687 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368324    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvh4b\" (UniqueName: \"kubernetes.io/projected/96f33c73-db06-4ed5-87c4-8e5b7def9a49-kube-api-access-pvh4b\") pod \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") "
Sep 17 19:27:06.368687 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368350    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-tmp-dir\") pod \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") "
Sep 17 19:27:06.368687 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368377    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-model-cache\") pod \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") "
Sep 17 19:27:06.368687 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368410    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96f33c73-db06-4ed5-87c4-8e5b7def9a49-tls-certs\") pod \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") "
Sep 17 19:27:06.368687 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368447    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-kserve-provision-location\") pod \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") "
Sep 17 19:27:06.368687 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368472    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hssql\" (UniqueName: \"kubernetes.io/projected/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-kube-api-access-hssql\") pod \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\" (UID: \"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4\") "
Sep 17 19:27:06.368687 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368506    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-kserve-provision-location\") pod \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") "
Sep 17 19:27:06.368687 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368518    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" (UID: "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:06.368687 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368557    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-home\") pod \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\" (UID: \"96f33c73-db06-4ed5-87c4-8e5b7def9a49\") "
Sep 17 19:27:06.369179 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368761    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-model-cache" (OuterVolumeSpecName: "model-cache") pod "96f33c73-db06-4ed5-87c4-8e5b7def9a49" (UID: "96f33c73-db06-4ed5-87c4-8e5b7def9a49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:06.369179 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368837    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.369179 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368856    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.369179 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.368831    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f33c73-db06-4ed5-87c4-8e5b7def9a49-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "96f33c73-db06-4ed5-87c4-8e5b7def9a49" (UID: "96f33c73-db06-4ed5-87c4-8e5b7def9a49"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:06.370383 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.370105    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-home" (OuterVolumeSpecName: "home") pod "96f33c73-db06-4ed5-87c4-8e5b7def9a49" (UID: "96f33c73-db06-4ed5-87c4-8e5b7def9a49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:06.371101 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.371076    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" (UID: "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:06.371760 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.371732    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-home" (OuterVolumeSpecName: "home") pod "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" (UID: "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:06.372190 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.372148    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" (UID: "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:06.372490 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.372367    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-dshm" (OuterVolumeSpecName: "dshm") pod "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" (UID: "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:06.372490 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.372453    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f33c73-db06-4ed5-87c4-8e5b7def9a49-kube-api-access-pvh4b" (OuterVolumeSpecName: "kube-api-access-pvh4b") pod "96f33c73-db06-4ed5-87c4-8e5b7def9a49" (UID: "96f33c73-db06-4ed5-87c4-8e5b7def9a49"). InnerVolumeSpecName "kube-api-access-pvh4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:06.372752 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.372723    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-dshm" (OuterVolumeSpecName: "dshm") pod "96f33c73-db06-4ed5-87c4-8e5b7def9a49" (UID: "96f33c73-db06-4ed5-87c4-8e5b7def9a49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:06.372861 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.372840    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-kube-api-access-hssql" (OuterVolumeSpecName: "kube-api-access-hssql") pod "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" (UID: "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4"). InnerVolumeSpecName "kube-api-access-hssql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:06.374227 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.374208    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96f33c73-db06-4ed5-87c4-8e5b7def9a49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96f33c73-db06-4ed5-87c4-8e5b7def9a49" (UID: "96f33c73-db06-4ed5-87c4-8e5b7def9a49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:06.380375 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.380340    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96f33c73-db06-4ed5-87c4-8e5b7def9a49" (UID: "96f33c73-db06-4ed5-87c4-8e5b7def9a49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:06.386800 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.386675    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" (UID: "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:06.390156 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.390117    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96f33c73-db06-4ed5-87c4-8e5b7def9a49" (UID: "96f33c73-db06-4ed5-87c4-8e5b7def9a49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:06.390398 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.390372    2587 generic.go:356] "Generic (PLEG): container finished" podID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerID="f484896164bbf3d6f47810370e1e0d169213e4e1781042bcf76fe5f63929a529" exitCode=0
Sep 17 19:27:06.390499 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.390402    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" event={"ID":"96f33c73-db06-4ed5-87c4-8e5b7def9a49","Type":"ContainerDied","Data":"f484896164bbf3d6f47810370e1e0d169213e4e1781042bcf76fe5f63929a529"}
Sep 17 19:27:06.390499 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.390435    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"
Sep 17 19:27:06.390499 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.390454    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx" event={"ID":"96f33c73-db06-4ed5-87c4-8e5b7def9a49","Type":"ContainerDied","Data":"cf11a5c09eb22b3dfefc38bb882e336942220b8c40b3b777a43e5fe923cf845e"}
Sep 17 19:27:06.390499 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.390378    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" (UID: "7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:06.390499 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.390475    2587 scope.go:117] "RemoveContainer" containerID="f484896164bbf3d6f47810370e1e0d169213e4e1781042bcf76fe5f63929a529"
Sep 17 19:27:06.393139 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.393111    2587 generic.go:356] "Generic (PLEG): container finished" podID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerID="f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5" exitCode=0
Sep 17 19:27:06.393139 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.393138    2587 generic.go:356] "Generic (PLEG): container finished" podID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerID="7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2" exitCode=0
Sep 17 19:27:06.393299 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.393191    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"
Sep 17 19:27:06.393299 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.393191    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" event={"ID":"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4","Type":"ContainerDied","Data":"f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5"}
Sep 17 19:27:06.393378 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.393342    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" event={"ID":"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4","Type":"ContainerDied","Data":"7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2"}
Sep 17 19:27:06.393419 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.393392    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv" event={"ID":"7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4","Type":"ContainerDied","Data":"f06921b3eae8f2f35a6f8a22838a518420e054793dbcffd4091709832a4f7515"}
Sep 17 19:27:06.402466 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.402437    2587 scope.go:117] "RemoveContainer" containerID="a04d459c028c5c99aa12c42dbfc0532817d799bc4e78a3373125f28ad4601177"
Sep 17 19:27:06.411054 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.411033    2587 scope.go:117] "RemoveContainer" containerID="f484896164bbf3d6f47810370e1e0d169213e4e1781042bcf76fe5f63929a529"
Sep 17 19:27:06.411379 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:27:06.411361    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f484896164bbf3d6f47810370e1e0d169213e4e1781042bcf76fe5f63929a529\": container with ID starting with f484896164bbf3d6f47810370e1e0d169213e4e1781042bcf76fe5f63929a529 not found: ID does not exist" containerID="f484896164bbf3d6f47810370e1e0d169213e4e1781042bcf76fe5f63929a529"
Sep 17 19:27:06.411462 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.411390    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f484896164bbf3d6f47810370e1e0d169213e4e1781042bcf76fe5f63929a529"} err="failed to get container status \"f484896164bbf3d6f47810370e1e0d169213e4e1781042bcf76fe5f63929a529\": rpc error: code = NotFound desc = could not find container \"f484896164bbf3d6f47810370e1e0d169213e4e1781042bcf76fe5f63929a529\": container with ID starting with f484896164bbf3d6f47810370e1e0d169213e4e1781042bcf76fe5f63929a529 not found: ID does not exist"
Sep 17 19:27:06.411462 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.411414    2587 scope.go:117] "RemoveContainer" containerID="a04d459c028c5c99aa12c42dbfc0532817d799bc4e78a3373125f28ad4601177"
Sep 17 19:27:06.411693 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:27:06.411673    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a04d459c028c5c99aa12c42dbfc0532817d799bc4e78a3373125f28ad4601177\": container with ID starting with a04d459c028c5c99aa12c42dbfc0532817d799bc4e78a3373125f28ad4601177 not found: ID does not exist" containerID="a04d459c028c5c99aa12c42dbfc0532817d799bc4e78a3373125f28ad4601177"
Sep 17 19:27:06.411739 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.411699    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04d459c028c5c99aa12c42dbfc0532817d799bc4e78a3373125f28ad4601177"} err="failed to get container status \"a04d459c028c5c99aa12c42dbfc0532817d799bc4e78a3373125f28ad4601177\": rpc error: code = NotFound desc = could not find container \"a04d459c028c5c99aa12c42dbfc0532817d799bc4e78a3373125f28ad4601177\": container with ID starting with a04d459c028c5c99aa12c42dbfc0532817d799bc4e78a3373125f28ad4601177 not found: ID does not exist"
Sep 17 19:27:06.411739 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.411715    2587 scope.go:117] "RemoveContainer" containerID="f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5"
Sep 17 19:27:06.417882 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.417856    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"]
Sep 17 19:27:06.421556 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.421065    2587 scope.go:117] "RemoveContainer" containerID="517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71"
Sep 17 19:27:06.424282 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.424250    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74p2hqx"]
Sep 17 19:27:06.431694 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.431671    2587 scope.go:117] "RemoveContainer" containerID="7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2"
Sep 17 19:27:06.437545 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.437517    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"]
Sep 17 19:27:06.440509 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.440491    2587 scope.go:117] "RemoveContainer" containerID="f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5"
Sep 17 19:27:06.440967 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:27:06.440923    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5\": container with ID starting with f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5 not found: ID does not exist" containerID="f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5"
Sep 17 19:27:06.441089 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.440981    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5"} err="failed to get container status \"f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5\": rpc error: code = NotFound desc = could not find container \"f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5\": container with ID starting with f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5 not found: ID does not exist"
Sep 17 19:27:06.441089 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.441010    2587 scope.go:117] "RemoveContainer" containerID="517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71"
Sep 17 19:27:06.441431 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:27:06.441392    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71\": container with ID starting with 517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71 not found: ID does not exist" containerID="517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71"
Sep 17 19:27:06.441537 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.441426    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71"} err="failed to get container status \"517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71\": rpc error: code = NotFound desc = could not find container \"517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71\": container with ID starting with 517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71 not found: ID does not exist"
Sep 17 19:27:06.441537 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.441448    2587 scope.go:117] "RemoveContainer" containerID="7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2"
Sep 17 19:27:06.441854 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:27:06.441823    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2\": container with ID starting with 7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2 not found: ID does not exist" containerID="7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2"
Sep 17 19:27:06.442057 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.442019    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2"} err="failed to get container status \"7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2\": rpc error: code = NotFound desc = could not find container \"7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2\": container with ID starting with 7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2 not found: ID does not exist"
Sep 17 19:27:06.442057 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.442058    2587 scope.go:117] "RemoveContainer" containerID="f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5"
Sep 17 19:27:06.442355 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.442332    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5"} err="failed to get container status \"f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5\": rpc error: code = NotFound desc = could not find container \"f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5\": container with ID starting with f9bbaf4d15fc2c31022e4dbbda58fbab2be39e46f04360ac2f362ef8be0a22f5 not found: ID does not exist"
Sep 17 19:27:06.442433 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.442356    2587 scope.go:117] "RemoveContainer" containerID="517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71"
Sep 17 19:27:06.442618 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.442598    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71"} err="failed to get container status \"517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71\": rpc error: code = NotFound desc = could not find container \"517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71\": container with ID starting with 517950ffc27fb994166632f3bd72056f610629b9994727e0d54e0e5627444d71 not found: ID does not exist"
Sep 17 19:27:06.442677 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.442620    2587 scope.go:117] "RemoveContainer" containerID="7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2"
Sep 17 19:27:06.442857 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.442833    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2"} err="failed to get container status \"7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2\": rpc error: code = NotFound desc = could not find container \"7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2\": container with ID starting with 7f5c7e2579d3f7e34233706d6850e5d7260cd1a626c023ab77432acea13190f2 not found: ID does not exist"
Sep 17 19:27:06.443337 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.443315    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6b99bdc6872wwgv"]
Sep 17 19:27:06.470216 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.470181    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.470216 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.470211    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.470216 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.470220    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvh4b\" (UniqueName: \"kubernetes.io/projected/96f33c73-db06-4ed5-87c4-8e5b7def9a49-kube-api-access-pvh4b\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.470440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.470228    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.470440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.470240    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96f33c73-db06-4ed5-87c4-8e5b7def9a49-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.470440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.470248    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.470440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.470256    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hssql\" (UniqueName: \"kubernetes.io/projected/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-kube-api-access-hssql\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.470440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.470264    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.470440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.470272    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.470440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.470281    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.470440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.470289    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96f33c73-db06-4ed5-87c4-8e5b7def9a49-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.470440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.470296    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.470440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.470303    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96f33c73-db06-4ed5-87c4-8e5b7def9a49-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:06.470440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:06.470311    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:27:08.257338 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:08.257307    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" path="/var/lib/kubelet/pods/7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4/volumes"
Sep 17 19:27:08.257834 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:08.257820    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" path="/var/lib/kubelet/pods/96f33c73-db06-4ed5-87c4-8e5b7def9a49/volumes"
Sep 17 19:27:09.667160 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:09.667125    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:09.807212 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:09.807180    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:09.937656 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:09.937575    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:16.152705 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:16.152671    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:19.015543 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.015507    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"]
Sep 17 19:27:19.016005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.015808    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="storage-initializer"
Sep 17 19:27:19.016005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.015819    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="storage-initializer"
Sep 17 19:27:19.016005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.015826    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main"
Sep 17 19:27:19.016005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.015832    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main"
Sep 17 19:27:19.016005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.015839    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="llm-d-routing-sidecar"
Sep 17 19:27:19.016005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.015844    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="llm-d-routing-sidecar"
Sep 17 19:27:19.016005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.015857    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main"
Sep 17 19:27:19.016005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.015862    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main"
Sep 17 19:27:19.016005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.015871    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="storage-initializer"
Sep 17 19:27:19.016005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.015876    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="storage-initializer"
Sep 17 19:27:19.016005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.015932    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="main"
Sep 17 19:27:19.016005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.015956    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d4ac10f-5bd3-431f-a2a1-6b6efd9daea4" containerName="llm-d-routing-sidecar"
Sep 17 19:27:19.016005 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.015965    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="96f33c73-db06-4ed5-87c4-8e5b7def9a49" containerName="main"
Sep 17 19:27:19.020365 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.020343    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.024783 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.024758    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 17 19:27:19.025213 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.025193    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 17 19:27:19.025387 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.025375    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:27:19.025472 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.025451    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 17 19:27:19.025530 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.025502    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-j4xv2\""
Sep 17 19:27:19.025833 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.025819    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 17 19:27:19.041579 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.041545    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"]
Sep 17 19:27:19.055464 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.055432    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:19.067564 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.067530    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-dshm\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.067745 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.067571    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkk2s\" (UniqueName: \"kubernetes.io/projected/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-kube-api-access-pkk2s\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.067745 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.067648    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.067745 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.067682    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-tmp-dir\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.067745 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.067712    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-tls-certs\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.067745 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.067740    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-model-cache\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.068007 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.067789    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-home\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.068007 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.067835    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.168973 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.168901    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.169196 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.168992    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-dshm\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.169196 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.169019    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkk2s\" (UniqueName: \"kubernetes.io/projected/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-kube-api-access-pkk2s\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.169196 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.169062    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.169196 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.169180    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-tmp-dir\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.169442 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.169234    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-tls-certs\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.169442 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.169272    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-model-cache\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.169442 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.169308    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-home\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.169613 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.169498    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.169613 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.169596    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-model-cache\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.169728 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.169654    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-home\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.169788 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.169754    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-tmp-dir\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.169842 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.169811    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.171650 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.171624    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-dshm\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.171971 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.171929    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-tls-certs\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.179367 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.179335    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkk2s\" (UniqueName: \"kubernetes.io/projected/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-kube-api-access-pkk2s\") pod \"lora-single-adapter-test-kserve-5845d5c7b5-n7fnn\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.330867 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.330776    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:19.622732 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:19.622703    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"]
Sep 17 19:27:19.624439 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:27:19.624413    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86a4b3f_3d8a_41ac_adc0_8a73c26bdc6e.slice/crio-90e342b25617eedf19d4f76110f571b2a7d253773356b03c86abe2ca417912ea WatchSource:0}: Error finding container 90e342b25617eedf19d4f76110f571b2a7d253773356b03c86abe2ca417912ea: Status 404 returned error can't find the container with id 90e342b25617eedf19d4f76110f571b2a7d253773356b03c86abe2ca417912ea
Sep 17 19:27:20.453320 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:20.453275    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" event={"ID":"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e","Type":"ContainerStarted","Data":"8ce7a2207a7d69c2724d312d30c2afcc3f99c8bce0b19ae98e1133187830ea9e"}
Sep 17 19:27:20.453837 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:20.453338    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" event={"ID":"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e","Type":"ContainerStarted","Data":"90e342b25617eedf19d4f76110f571b2a7d253773356b03c86abe2ca417912ea"}
Sep 17 19:27:22.189866 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:22.189817    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:22.347309 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:22.347272    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:23.470168 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:23.470135    2587 generic.go:356] "Generic (PLEG): container finished" podID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerID="8ce7a2207a7d69c2724d312d30c2afcc3f99c8bce0b19ae98e1133187830ea9e" exitCode=0
Sep 17 19:27:23.470570 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:23.470210    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" event={"ID":"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e","Type":"ContainerDied","Data":"8ce7a2207a7d69c2724d312d30c2afcc3f99c8bce0b19ae98e1133187830ea9e"}
Sep 17 19:27:24.476504 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:24.476466    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" event={"ID":"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e","Type":"ContainerStarted","Data":"09eb1838167cd7a4f705b753acf9787545fca970118345ab0e45cb90bed0233e"}
Sep 17 19:27:24.501820 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:24.501761    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" podStartSLOduration=3.66722927 podStartE2EDuration="6.501743033s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.834513763s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:27:24.500155963 +0000 UTC m=+2896.379124721" watchObservedRunningTime="2026-09-17 19:27:24.501743033 +0000 UTC m=+2896.380711790"
Sep 17 19:27:25.331225 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:25.331191    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:29.331669 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:29.331622    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:27:29.333260 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:29.333228    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" podUID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 17 19:27:39.332267 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:39.332215    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" podUID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 17 19:27:39.726675 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:39.726588    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:40.447368 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:40.447328    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:40.541054 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:40.541014    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:46.539692 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:46.539647    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:47.559613 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.559579    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"]
Sep 17 19:27:47.561983 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.561960    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.564627 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.564600    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 19:27:47.564779 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.564659    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 19:27:47.565057 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.565039    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:27:47.565653 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.565631    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 19:27:47.566195 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.566179    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-5zmls\""
Sep 17 19:27:47.566742 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.566722    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 19:27:47.583080 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.583050    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"]
Sep 17 19:27:47.613621 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.613583    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:47.710329 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.710283    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.710506 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.710337    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.710506 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.710381    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cb41e18-93e7-45e4-a846-6398c1769cf4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.710506 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.710428    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.710506 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.710465    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpmlk\" (UniqueName: \"kubernetes.io/projected/7cb41e18-93e7-45e4-a846-6398c1769cf4-kube-api-access-rpmlk\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.710649 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.710505    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.710649 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.710561    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cb41e18-93e7-45e4-a846-6398c1769cf4-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.710649 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.710591    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.811385 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.811280    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.811385 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.811324    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.811385 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.811362    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cb41e18-93e7-45e4-a846-6398c1769cf4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.811678 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.811406    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.811678 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.811439    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpmlk\" (UniqueName: \"kubernetes.io/projected/7cb41e18-93e7-45e4-a846-6398c1769cf4-kube-api-access-rpmlk\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.811678 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.811469    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.811678 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.811528    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cb41e18-93e7-45e4-a846-6398c1769cf4-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.811678 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.811556    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.811966 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.811926    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.812037 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.812003    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.812227 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.812205    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.812598 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.812573    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cb41e18-93e7-45e4-a846-6398c1769cf4-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.812694 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.812669    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.814077 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.814050    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.814203 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.814176    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cb41e18-93e7-45e4-a846-6398c1769cf4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.824360 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.824334    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpmlk\" (UniqueName: \"kubernetes.io/projected/7cb41e18-93e7-45e4-a846-6398c1769cf4-kube-api-access-rpmlk\") pod \"custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:47.826534 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.826509    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:47.874595 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:47.874554    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:48.155416 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:48.155381    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"]
Sep 17 19:27:48.157321 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:27:48.157297    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb41e18_93e7_45e4_a846_6398c1769cf4.slice/crio-edcafd9f22218ab64d8a9110bc138964b1dc1cf50080ec3841c85bd1021fd17b WatchSource:0}: Error finding container edcafd9f22218ab64d8a9110bc138964b1dc1cf50080ec3841c85bd1021fd17b: Status 404 returned error can't find the container with id edcafd9f22218ab64d8a9110bc138964b1dc1cf50080ec3841c85bd1021fd17b
Sep 17 19:27:48.586210 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:48.586175    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" event={"ID":"7cb41e18-93e7-45e4-a846-6398c1769cf4","Type":"ContainerStarted","Data":"b67d03326149ff421a06f1f44054e8f5b822a2617056ba0fcf7f81cb20328f16"}
Sep 17 19:27:48.586627 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:48.586234    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" event={"ID":"7cb41e18-93e7-45e4-a846-6398c1769cf4","Type":"ContainerStarted","Data":"edcafd9f22218ab64d8a9110bc138964b1dc1cf50080ec3841c85bd1021fd17b"}
Sep 17 19:27:49.332063 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:49.332022    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" podUID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 17 19:27:50.598996 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:50.598956    2587 generic.go:356] "Generic (PLEG): container finished" podID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerID="b67d03326149ff421a06f1f44054e8f5b822a2617056ba0fcf7f81cb20328f16" exitCode=0
Sep 17 19:27:50.599494 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:50.598969    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" event={"ID":"7cb41e18-93e7-45e4-a846-6398c1769cf4","Type":"ContainerDied","Data":"b67d03326149ff421a06f1f44054e8f5b822a2617056ba0fcf7f81cb20328f16"}
Sep 17 19:27:51.606873 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:51.606839    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" event={"ID":"7cb41e18-93e7-45e4-a846-6398c1769cf4","Type":"ContainerStarted","Data":"b97ceffa270f22f358bf037369062830c8c37b69131df9a05fa07300d643803c"}
Sep 17 19:27:51.635420 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:51.635369    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" podStartSLOduration=2.294822327 podStartE2EDuration="4.635349467s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.34052714s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:27:51.633805815 +0000 UTC m=+2923.512774586" watchObservedRunningTime="2026-09-17 19:27:51.635349467 +0000 UTC m=+2923.514318228"
Sep 17 19:27:52.260603 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:52.260570    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:52.330099 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:52.330061    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:57.874911 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:57.874877    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:57.875433 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:57.874970    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:27:57.876766 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:57.876738    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:27:59.332007 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:27:59.331926    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" podUID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 17 19:28:03.030774 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:03.030733    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:07.875961 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:07.875902    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:28:08.092980 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:08.092927    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:08.183853 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:08.183759    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:09.331683 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:09.331626    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" podUID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 17 19:28:17.875458 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:17.875408    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:28:19.332232 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:19.332175    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" podUID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 17 19:28:22.366240 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:22.366148    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:22.467149 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:22.467118    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:27.875618 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:27.875567    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:28:29.332138 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:29.332080    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" podUID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 17 19:28:37.875383 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:37.875334    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:28:39.342301 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:39.342263    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:28:39.350174 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:39.350141    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:28:47.875800 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:47.875748    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:28:57.875670 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:28:57.875627    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:29:05.726093 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:05.726054    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-5845d5c7b5-n7fnn_c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e/storage-initializer/0.log"
Sep 17 19:29:05.736647 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:05.736615    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-5845d5c7b5-n7fnn_c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e/main/0.log"
Sep 17 19:29:07.334209 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:07.334172    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"]
Sep 17 19:29:07.375335 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:07.375299    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:07.876026 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:07.875983    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:29:10.400029 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:10.399996    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:29:10.404490 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:10.404463    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:29:17.875370 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:17.875322    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:29:22.357342 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:22.357303    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" podUID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerName="main" containerID="cri-o://09eb1838167cd7a4f705b753acf9787545fca970118345ab0e45cb90bed0233e" gracePeriod=45
Sep 17 19:29:23.310781 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.310750    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:29:23.382758 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.382658    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-model-cache\") pod \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") "
Sep 17 19:29:23.382758 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.382697    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkk2s\" (UniqueName: \"kubernetes.io/projected/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-kube-api-access-pkk2s\") pod \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") "
Sep 17 19:29:23.382758 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.382728    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-cabundle-cert\") pod \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") "
Sep 17 19:29:23.382758 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.382751    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-home\") pod \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") "
Sep 17 19:29:23.383460 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.382815    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-tls-certs\") pod \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") "
Sep 17 19:29:23.383460 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.382861    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-tmp-dir\") pod \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") "
Sep 17 19:29:23.383460 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.382910    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-dshm\") pod \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") "
Sep 17 19:29:23.383460 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.382963    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-kserve-provision-location\") pod \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\" (UID: \"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e\") "
Sep 17 19:29:23.383460 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.383252    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" (UID: "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:29:23.383460 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.383286    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-home" (OuterVolumeSpecName: "home") pod "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" (UID: "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:23.385648 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.385612    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" (UID: "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:29:23.386187 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.386159    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" (UID: "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:23.387494 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.387455    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-dshm" (OuterVolumeSpecName: "dshm") pod "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" (UID: "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:23.387810 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.387788    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-kube-api-access-pkk2s" (OuterVolumeSpecName: "kube-api-access-pkk2s") pod "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" (UID: "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e"). InnerVolumeSpecName "kube-api-access-pkk2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:29:23.395403 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.395368    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" (UID: "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:23.411381 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.411322    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" (UID: "c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:23.484459 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.484419    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:29:23.484459 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.484452    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:29:23.484459 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.484459    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:29:23.484459 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.484467    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:29:23.484717 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.484474    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:29:23.484717 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.484482    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:29:23.484717 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.484491    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:29:23.484717 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:23.484499    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkk2s\" (UniqueName: \"kubernetes.io/projected/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e-kube-api-access-pkk2s\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:29:24.068226 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:24.068187    2587 generic.go:356] "Generic (PLEG): container finished" podID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerID="09eb1838167cd7a4f705b753acf9787545fca970118345ab0e45cb90bed0233e" exitCode=0
Sep 17 19:29:24.068442 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:24.068240    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"
Sep 17 19:29:24.068442 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:24.068274    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" event={"ID":"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e","Type":"ContainerDied","Data":"09eb1838167cd7a4f705b753acf9787545fca970118345ab0e45cb90bed0233e"}
Sep 17 19:29:24.068442 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:24.068330    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn" event={"ID":"c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e","Type":"ContainerDied","Data":"90e342b25617eedf19d4f76110f571b2a7d253773356b03c86abe2ca417912ea"}
Sep 17 19:29:24.068442 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:24.068358    2587 scope.go:117] "RemoveContainer" containerID="09eb1838167cd7a4f705b753acf9787545fca970118345ab0e45cb90bed0233e"
Sep 17 19:29:24.077788 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:24.077767    2587 scope.go:117] "RemoveContainer" containerID="8ce7a2207a7d69c2724d312d30c2afcc3f99c8bce0b19ae98e1133187830ea9e"
Sep 17 19:29:24.086722 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:24.086699    2587 scope.go:117] "RemoveContainer" containerID="09eb1838167cd7a4f705b753acf9787545fca970118345ab0e45cb90bed0233e"
Sep 17 19:29:24.087104 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:29:24.087085    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09eb1838167cd7a4f705b753acf9787545fca970118345ab0e45cb90bed0233e\": container with ID starting with 09eb1838167cd7a4f705b753acf9787545fca970118345ab0e45cb90bed0233e not found: ID does not exist" containerID="09eb1838167cd7a4f705b753acf9787545fca970118345ab0e45cb90bed0233e"
Sep 17 19:29:24.087200 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:24.087121    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09eb1838167cd7a4f705b753acf9787545fca970118345ab0e45cb90bed0233e"} err="failed to get container status \"09eb1838167cd7a4f705b753acf9787545fca970118345ab0e45cb90bed0233e\": rpc error: code = NotFound desc = could not find container \"09eb1838167cd7a4f705b753acf9787545fca970118345ab0e45cb90bed0233e\": container with ID starting with 09eb1838167cd7a4f705b753acf9787545fca970118345ab0e45cb90bed0233e not found: ID does not exist"
Sep 17 19:29:24.087200 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:24.087141    2587 scope.go:117] "RemoveContainer" containerID="8ce7a2207a7d69c2724d312d30c2afcc3f99c8bce0b19ae98e1133187830ea9e"
Sep 17 19:29:24.087423 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:29:24.087403    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce7a2207a7d69c2724d312d30c2afcc3f99c8bce0b19ae98e1133187830ea9e\": container with ID starting with 8ce7a2207a7d69c2724d312d30c2afcc3f99c8bce0b19ae98e1133187830ea9e not found: ID does not exist" containerID="8ce7a2207a7d69c2724d312d30c2afcc3f99c8bce0b19ae98e1133187830ea9e"
Sep 17 19:29:24.087482 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:24.087435    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce7a2207a7d69c2724d312d30c2afcc3f99c8bce0b19ae98e1133187830ea9e"} err="failed to get container status \"8ce7a2207a7d69c2724d312d30c2afcc3f99c8bce0b19ae98e1133187830ea9e\": rpc error: code = NotFound desc = could not find container \"8ce7a2207a7d69c2724d312d30c2afcc3f99c8bce0b19ae98e1133187830ea9e\": container with ID starting with 8ce7a2207a7d69c2724d312d30c2afcc3f99c8bce0b19ae98e1133187830ea9e not found: ID does not exist"
Sep 17 19:29:24.115769 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:24.115729    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"]
Sep 17 19:29:24.126185 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:24.126154    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5845d5c7b5-n7fnn"]
Sep 17 19:29:24.258032 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:24.257996    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" path="/var/lib/kubelet/pods/c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e/volumes"
Sep 17 19:29:27.423291 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:27.423257    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:27.530818 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:27.530783    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:27.625607 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:27.625548    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:27.875474 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:27.875418    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:29:33.694400 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:33.694362    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:33.801332 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:33.801298    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:37.876191 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:37.876140    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:29:38.179326 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.179245    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"]
Sep 17 19:29:38.179588 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.179574    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerName="storage-initializer"
Sep 17 19:29:38.179640 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.179590    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerName="storage-initializer"
Sep 17 19:29:38.179640 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.179598    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerName="main"
Sep 17 19:29:38.179640 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.179604    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerName="main"
Sep 17 19:29:38.179734 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.179666    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="c86a4b3f-3d8a-41ac-adc0-8a73c26bdc6e" containerName="main"
Sep 17 19:29:38.186140 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.186118    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.189887 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.189865    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 17 19:29:38.190892 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.190867    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:29:38.191423 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.191401    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 17 19:29:38.191660 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.191643    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-q69b2\""
Sep 17 19:29:38.191734 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.191713    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 17 19:29:38.192278 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.192250    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 17 19:29:38.200729 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.200704    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"]
Sep 17 19:29:38.204812 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.204784    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:38.317892 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.317854    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-home\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.317892 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.317907    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-dshm\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.318161 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.317925    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-model-cache\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.318161 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.318014    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99fgr\" (UniqueName: \"kubernetes.io/projected/53484c95-a304-4408-89e2-4ea262687889-kube-api-access-99fgr\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.318161 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.318047    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.318161 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.318066    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53484c95-a304-4408-89e2-4ea262687889-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.318161 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.318082    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.318161 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.318102    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53484c95-a304-4408-89e2-4ea262687889-tls-certs\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.419317 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.419276    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.419521 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.419326    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53484c95-a304-4408-89e2-4ea262687889-tls-certs\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.419521 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.419380    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-home\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.419521 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.419411    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-dshm\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.419521 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.419433    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-model-cache\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.419521 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.419477    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99fgr\" (UniqueName: \"kubernetes.io/projected/53484c95-a304-4408-89e2-4ea262687889-kube-api-access-99fgr\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.419788 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.419727    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.419788 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.419775    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-home\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.419923 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.419869    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-model-cache\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.419923 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.419529    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.420042 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.419955    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53484c95-a304-4408-89e2-4ea262687889-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.420108 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.420088    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.420537 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.420520    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53484c95-a304-4408-89e2-4ea262687889-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.422133 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.422113    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53484c95-a304-4408-89e2-4ea262687889-tls-certs\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.422230 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.422213    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-dshm\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.430250 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.430185    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99fgr\" (UniqueName: \"kubernetes.io/projected/53484c95-a304-4408-89e2-4ea262687889-kube-api-access-99fgr\") pod \"lora-multiple-adapters-test-kserve-64484479fd-gxd99\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.497873 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.497826    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:38.795720 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:38.795688    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"]
Sep 17 19:29:38.798193 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:29:38.798151    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53484c95_a304_4408_89e2_4ea262687889.slice/crio-35847483c00ba4c239aa06b672ce1933e879bdae467ef75cbd92dc4bfbe5e104 WatchSource:0}: Error finding container 35847483c00ba4c239aa06b672ce1933e879bdae467ef75cbd92dc4bfbe5e104: Status 404 returned error can't find the container with id 35847483c00ba4c239aa06b672ce1933e879bdae467ef75cbd92dc4bfbe5e104
Sep 17 19:29:39.140673 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:39.140585    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" event={"ID":"53484c95-a304-4408-89e2-4ea262687889","Type":"ContainerStarted","Data":"cc967fbabec56c886f633ddd231d21a14e40c640741b6f7c27cc038e6a8c311c"}
Sep 17 19:29:39.140673 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:39.140642    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" event={"ID":"53484c95-a304-4408-89e2-4ea262687889","Type":"ContainerStarted","Data":"35847483c00ba4c239aa06b672ce1933e879bdae467ef75cbd92dc4bfbe5e104"}
Sep 17 19:29:42.157903 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:42.157866    2587 generic.go:356] "Generic (PLEG): container finished" podID="53484c95-a304-4408-89e2-4ea262687889" containerID="cc967fbabec56c886f633ddd231d21a14e40c640741b6f7c27cc038e6a8c311c" exitCode=0
Sep 17 19:29:42.158317 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:42.157977    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" event={"ID":"53484c95-a304-4408-89e2-4ea262687889","Type":"ContainerDied","Data":"cc967fbabec56c886f633ddd231d21a14e40c640741b6f7c27cc038e6a8c311c"}
Sep 17 19:29:43.168315 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:43.168269    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" event={"ID":"53484c95-a304-4408-89e2-4ea262687889","Type":"ContainerStarted","Data":"ff0615c69b9ac55d22d4f0c76f2d993328aa627c818ca669fd3aab80b4138d5d"}
Sep 17 19:29:43.194019 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:43.193915    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" podStartSLOduration=2.136203343 podStartE2EDuration="5.19388919s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.057685847s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:29:43.191823862 +0000 UTC m=+3035.070792674" watchObservedRunningTime="2026-09-17 19:29:43.19388919 +0000 UTC m=+3035.072857962"
Sep 17 19:29:43.498450 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:43.498413    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:47.885125 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:47.885084    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:29:47.893128 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:47.893081    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:29:48.498729 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:48.498694    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:29:48.500380 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:48.500350    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" podUID="53484c95-a304-4408-89e2-4ea262687889" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 19:29:58.499183 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:29:58.499053    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" podUID="53484c95-a304-4408-89e2-4ea262687889" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 19:30:08.499258 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:08.499215    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" podUID="53484c95-a304-4408-89e2-4ea262687889" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 19:30:12.194880 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:12.194846    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:12.228742 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:12.228704    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"]
Sep 17 19:30:12.320778 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:12.320744    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:18.499029 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:18.498975    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" podUID="53484c95-a304-4408-89e2-4ea262687889" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 19:30:27.251625 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:27.251569    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main" containerID="cri-o://b97ceffa270f22f358bf037369062830c8c37b69131df9a05fa07300d643803c" gracePeriod=45
Sep 17 19:30:27.886415 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:27.886374    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 17 19:30:28.006022 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.005994    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:30:28.055808 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.055720    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpmlk\" (UniqueName: \"kubernetes.io/projected/7cb41e18-93e7-45e4-a846-6398c1769cf4-kube-api-access-rpmlk\") pod \"7cb41e18-93e7-45e4-a846-6398c1769cf4\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") "
Sep 17 19:30:28.055808 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.055764    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-kserve-provision-location\") pod \"7cb41e18-93e7-45e4-a846-6398c1769cf4\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") "
Sep 17 19:30:28.055808 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.055802    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-tmp-dir\") pod \"7cb41e18-93e7-45e4-a846-6398c1769cf4\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") "
Sep 17 19:30:28.056137 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.055837    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-home\") pod \"7cb41e18-93e7-45e4-a846-6398c1769cf4\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") "
Sep 17 19:30:28.056137 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.055864    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cb41e18-93e7-45e4-a846-6398c1769cf4-cabundle-cert\") pod \"7cb41e18-93e7-45e4-a846-6398c1769cf4\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") "
Sep 17 19:30:28.056137 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.055901    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-dshm\") pod \"7cb41e18-93e7-45e4-a846-6398c1769cf4\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") "
Sep 17 19:30:28.056395 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.056367    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb41e18-93e7-45e4-a846-6398c1769cf4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7cb41e18-93e7-45e4-a846-6398c1769cf4" (UID: "7cb41e18-93e7-45e4-a846-6398c1769cf4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:30:28.056780 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.056757    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-home" (OuterVolumeSpecName: "home") pod "7cb41e18-93e7-45e4-a846-6398c1769cf4" (UID: "7cb41e18-93e7-45e4-a846-6398c1769cf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:28.059051 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.059013    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb41e18-93e7-45e4-a846-6398c1769cf4-kube-api-access-rpmlk" (OuterVolumeSpecName: "kube-api-access-rpmlk") pod "7cb41e18-93e7-45e4-a846-6398c1769cf4" (UID: "7cb41e18-93e7-45e4-a846-6398c1769cf4"). InnerVolumeSpecName "kube-api-access-rpmlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:30:28.059195 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.059093    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-dshm" (OuterVolumeSpecName: "dshm") pod "7cb41e18-93e7-45e4-a846-6398c1769cf4" (UID: "7cb41e18-93e7-45e4-a846-6398c1769cf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:28.074917 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.074880    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7cb41e18-93e7-45e4-a846-6398c1769cf4" (UID: "7cb41e18-93e7-45e4-a846-6398c1769cf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:28.079588 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.079562    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7cb41e18-93e7-45e4-a846-6398c1769cf4" (UID: "7cb41e18-93e7-45e4-a846-6398c1769cf4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:28.156690 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.156657    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-model-cache\") pod \"7cb41e18-93e7-45e4-a846-6398c1769cf4\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") "
Sep 17 19:30:28.156841 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.156700    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cb41e18-93e7-45e4-a846-6398c1769cf4-tls-certs\") pod \"7cb41e18-93e7-45e4-a846-6398c1769cf4\" (UID: \"7cb41e18-93e7-45e4-a846-6398c1769cf4\") "
Sep 17 19:30:28.156887 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.156852    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.156887 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.156883    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.156971 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.156892    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.156971 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.156900    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cb41e18-93e7-45e4-a846-6398c1769cf4-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.156971 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.156908    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.156971 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.156915    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpmlk\" (UniqueName: \"kubernetes.io/projected/7cb41e18-93e7-45e4-a846-6398c1769cf4-kube-api-access-rpmlk\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.156971 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.156924    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-model-cache" (OuterVolumeSpecName: "model-cache") pod "7cb41e18-93e7-45e4-a846-6398c1769cf4" (UID: "7cb41e18-93e7-45e4-a846-6398c1769cf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:28.159056 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.159026    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cb41e18-93e7-45e4-a846-6398c1769cf4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7cb41e18-93e7-45e4-a846-6398c1769cf4" (UID: "7cb41e18-93e7-45e4-a846-6398c1769cf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:30:28.257475 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.257443    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cb41e18-93e7-45e4-a846-6398c1769cf4-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.257475 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.257470    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cb41e18-93e7-45e4-a846-6398c1769cf4-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:30:28.389452 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.389348    2587 generic.go:356] "Generic (PLEG): container finished" podID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerID="b97ceffa270f22f358bf037369062830c8c37b69131df9a05fa07300d643803c" exitCode=0
Sep 17 19:30:28.389452 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.389407    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"
Sep 17 19:30:28.389452 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.389436    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" event={"ID":"7cb41e18-93e7-45e4-a846-6398c1769cf4","Type":"ContainerDied","Data":"b97ceffa270f22f358bf037369062830c8c37b69131df9a05fa07300d643803c"}
Sep 17 19:30:28.389740 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.389493    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq" event={"ID":"7cb41e18-93e7-45e4-a846-6398c1769cf4","Type":"ContainerDied","Data":"edcafd9f22218ab64d8a9110bc138964b1dc1cf50080ec3841c85bd1021fd17b"}
Sep 17 19:30:28.389740 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.389515    2587 scope.go:117] "RemoveContainer" containerID="b97ceffa270f22f358bf037369062830c8c37b69131df9a05fa07300d643803c"
Sep 17 19:30:28.400127 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.400101    2587 scope.go:117] "RemoveContainer" containerID="b67d03326149ff421a06f1f44054e8f5b822a2617056ba0fcf7f81cb20328f16"
Sep 17 19:30:28.408983 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.408957    2587 scope.go:117] "RemoveContainer" containerID="b97ceffa270f22f358bf037369062830c8c37b69131df9a05fa07300d643803c"
Sep 17 19:30:28.409389 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:30:28.409365    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b97ceffa270f22f358bf037369062830c8c37b69131df9a05fa07300d643803c\": container with ID starting with b97ceffa270f22f358bf037369062830c8c37b69131df9a05fa07300d643803c not found: ID does not exist" containerID="b97ceffa270f22f358bf037369062830c8c37b69131df9a05fa07300d643803c"
Sep 17 19:30:28.409458 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.409398    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b97ceffa270f22f358bf037369062830c8c37b69131df9a05fa07300d643803c"} err="failed to get container status \"b97ceffa270f22f358bf037369062830c8c37b69131df9a05fa07300d643803c\": rpc error: code = NotFound desc = could not find container \"b97ceffa270f22f358bf037369062830c8c37b69131df9a05fa07300d643803c\": container with ID starting with b97ceffa270f22f358bf037369062830c8c37b69131df9a05fa07300d643803c not found: ID does not exist"
Sep 17 19:30:28.409458 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.409421    2587 scope.go:117] "RemoveContainer" containerID="b67d03326149ff421a06f1f44054e8f5b822a2617056ba0fcf7f81cb20328f16"
Sep 17 19:30:28.409539 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.409511    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"]
Sep 17 19:30:28.409751 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:30:28.409734    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67d03326149ff421a06f1f44054e8f5b822a2617056ba0fcf7f81cb20328f16\": container with ID starting with b67d03326149ff421a06f1f44054e8f5b822a2617056ba0fcf7f81cb20328f16 not found: ID does not exist" containerID="b67d03326149ff421a06f1f44054e8f5b822a2617056ba0fcf7f81cb20328f16"
Sep 17 19:30:28.409800 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.409758    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67d03326149ff421a06f1f44054e8f5b822a2617056ba0fcf7f81cb20328f16"} err="failed to get container status \"b67d03326149ff421a06f1f44054e8f5b822a2617056ba0fcf7f81cb20328f16\": rpc error: code = NotFound desc = could not find container \"b67d03326149ff421a06f1f44054e8f5b822a2617056ba0fcf7f81cb20328f16\": container with ID starting with b67d03326149ff421a06f1f44054e8f5b822a2617056ba0fcf7f81cb20328f16 not found: ID does not exist"
Sep 17 19:30:28.413441 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.413412    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-799676d578-vb9wq"]
Sep 17 19:30:28.498813 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:28.498773    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" podUID="53484c95-a304-4408-89e2-4ea262687889" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 19:30:30.258229 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:30.258195    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" path="/var/lib/kubelet/pods/7cb41e18-93e7-45e4-a846-6398c1769cf4/volumes"
Sep 17 19:30:38.498800 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:38.498752    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" podUID="53484c95-a304-4408-89e2-4ea262687889" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 19:30:48.498587 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:48.498534    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" podUID="53484c95-a304-4408-89e2-4ea262687889" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 19:30:58.508655 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:58.508617    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:30:58.542255 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:30:58.542221    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:31:02.241160 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:02.241119    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:02.328111 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:02.328067    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:02.517711 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:02.517620    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:02.618382 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:02.618352    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:06.503400 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:06.503370    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:09.062779 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:09.062746    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:11.472434 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.472398    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"]
Sep 17 19:31:11.472958 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.472703    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main"
Sep 17 19:31:11.472958 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.472714    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main"
Sep 17 19:31:11.472958 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.472723    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="storage-initializer"
Sep 17 19:31:11.472958 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.472730    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="storage-initializer"
Sep 17 19:31:11.472958 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.472802    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cb41e18-93e7-45e4-a846-6398c1769cf4" containerName="main"
Sep 17 19:31:11.476984 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.476960    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.480514 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.480490    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 19:31:11.480668 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.480488    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 19:31:11.480734 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.480662    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 19:31:11.481804 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.481784    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-lwnjk\""
Sep 17 19:31:11.481921 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.481838    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:31:11.481921 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.481784    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 19:31:11.494095 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.494066    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"]
Sep 17 19:31:11.505133 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.505093    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:11.634626 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.634581    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.634626 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.634630    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22462499-1df6-4fde-aec4-69b1657aed2e-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.634918 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.634696    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.634918 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.634748    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.634918 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.634781    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn8fs\" (UniqueName: \"kubernetes.io/projected/22462499-1df6-4fde-aec4-69b1657aed2e-kube-api-access-nn8fs\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.634918 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.634839    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-home\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.634918 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.634868    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.634918 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.634884    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22462499-1df6-4fde-aec4-69b1657aed2e-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.735932 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.735889    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22462499-1df6-4fde-aec4-69b1657aed2e-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.735932 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.735930    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.736229 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.735983    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.736229 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.736005    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn8fs\" (UniqueName: \"kubernetes.io/projected/22462499-1df6-4fde-aec4-69b1657aed2e-kube-api-access-nn8fs\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.736229 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.736033    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-home\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.736229 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.736056    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.736229 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.736078    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22462499-1df6-4fde-aec4-69b1657aed2e-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.736229 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.736109    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.736556 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.736429    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.736556 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.736507    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-home\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.736671 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.736578    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.736671 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.736648    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.736781 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.736767    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22462499-1df6-4fde-aec4-69b1657aed2e-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.738814 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.738785    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.738992 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.738975    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22462499-1df6-4fde-aec4-69b1657aed2e-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.744522 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.744495    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn8fs\" (UniqueName: \"kubernetes.io/projected/22462499-1df6-4fde-aec4-69b1657aed2e-kube-api-access-nn8fs\") pod \"router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:11.787557 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:11.787506    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:12.075368 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:12.075338    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"]
Sep 17 19:31:12.078383 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:31:12.078355    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22462499_1df6_4fde_aec4_69b1657aed2e.slice/crio-ad7079e8fc265f50cf6f72b8327a4dc55e1caeaa259182672c6077514931a89f WatchSource:0}: Error finding container ad7079e8fc265f50cf6f72b8327a4dc55e1caeaa259182672c6077514931a89f: Status 404 returned error can't find the container with id ad7079e8fc265f50cf6f72b8327a4dc55e1caeaa259182672c6077514931a89f
Sep 17 19:31:12.084852 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:12.084824    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:12.185105 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:12.185066    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:12.600454 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:12.600410    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" event={"ID":"22462499-1df6-4fde-aec4-69b1657aed2e","Type":"ContainerStarted","Data":"d34d344c8e34563bdf27c3827c12b1557798843da88016717e281db0f836c061"}
Sep 17 19:31:12.600977 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:12.600482    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" event={"ID":"22462499-1df6-4fde-aec4-69b1657aed2e","Type":"ContainerStarted","Data":"ad7079e8fc265f50cf6f72b8327a4dc55e1caeaa259182672c6077514931a89f"}
Sep 17 19:31:14.614330 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:14.614288    2587 generic.go:356] "Generic (PLEG): container finished" podID="22462499-1df6-4fde-aec4-69b1657aed2e" containerID="d34d344c8e34563bdf27c3827c12b1557798843da88016717e281db0f836c061" exitCode=0
Sep 17 19:31:14.614705 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:14.614363    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" event={"ID":"22462499-1df6-4fde-aec4-69b1657aed2e","Type":"ContainerDied","Data":"d34d344c8e34563bdf27c3827c12b1557798843da88016717e281db0f836c061"}
Sep 17 19:31:14.798775 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:14.798685    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:14.872670 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:14.872632    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:15.623193 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:15.623159    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" event={"ID":"22462499-1df6-4fde-aec4-69b1657aed2e","Type":"ContainerStarted","Data":"224630598a06224d811a825d39dee4fca6670af636a0a3b2e80a9973ce75874d"}
Sep 17 19:31:15.649635 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:15.649576    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" podStartSLOduration=2.255411581 podStartE2EDuration="4.649558079s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.394146498s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:31:15.648094775 +0000 UTC m=+3127.527063532" watchObservedRunningTime="2026-09-17 19:31:15.649558079 +0000 UTC m=+3127.528526835"
Sep 17 19:31:18.718034 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:18.717998    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:21.788479 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:21.788379    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:21.788479 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:21.788460    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:31:21.790160 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:21.790128    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 19:31:27.634702 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:27.634656    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-64484479fd-gxd99_53484c95-a304-4408-89e2-4ea262687889/storage-initializer/0.log"
Sep 17 19:31:27.644509 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:27.644480    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-64484479fd-gxd99_53484c95-a304-4408-89e2-4ea262687889/main/0.log"
Sep 17 19:31:28.707426 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:28.707387    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"]
Sep 17 19:31:28.726964 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:28.726904    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:31.788132 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:31.788092    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 19:31:41.788463 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:41.788404    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 19:31:43.731265 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:43.731202    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" podUID="53484c95-a304-4408-89e2-4ea262687889" containerName="main" containerID="cri-o://ff0615c69b9ac55d22d4f0c76f2d993328aa627c818ca669fd3aab80b4138d5d" gracePeriod=45
Sep 17 19:31:44.413122 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.413096    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:31:44.539513 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.539404    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-kserve-provision-location\") pod \"53484c95-a304-4408-89e2-4ea262687889\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") "
Sep 17 19:31:44.539711 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.539559    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-model-cache\") pod \"53484c95-a304-4408-89e2-4ea262687889\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") "
Sep 17 19:31:44.539711 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.539597    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53484c95-a304-4408-89e2-4ea262687889-cabundle-cert\") pod \"53484c95-a304-4408-89e2-4ea262687889\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") "
Sep 17 19:31:44.539711 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.539624    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-tmp-dir\") pod \"53484c95-a304-4408-89e2-4ea262687889\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") "
Sep 17 19:31:44.539711 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.539704    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-home\") pod \"53484c95-a304-4408-89e2-4ea262687889\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") "
Sep 17 19:31:44.539933 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.539731    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99fgr\" (UniqueName: \"kubernetes.io/projected/53484c95-a304-4408-89e2-4ea262687889-kube-api-access-99fgr\") pod \"53484c95-a304-4408-89e2-4ea262687889\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") "
Sep 17 19:31:44.539933 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.539786    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53484c95-a304-4408-89e2-4ea262687889-tls-certs\") pod \"53484c95-a304-4408-89e2-4ea262687889\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") "
Sep 17 19:31:44.539933 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.539825    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-dshm\") pod \"53484c95-a304-4408-89e2-4ea262687889\" (UID: \"53484c95-a304-4408-89e2-4ea262687889\") "
Sep 17 19:31:44.539933 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.539876    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-model-cache" (OuterVolumeSpecName: "model-cache") pod "53484c95-a304-4408-89e2-4ea262687889" (UID: "53484c95-a304-4408-89e2-4ea262687889"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:31:44.540186 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.539982    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53484c95-a304-4408-89e2-4ea262687889-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "53484c95-a304-4408-89e2-4ea262687889" (UID: "53484c95-a304-4408-89e2-4ea262687889"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:31:44.540186 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.540131    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:31:44.540356 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.540192    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-home" (OuterVolumeSpecName: "home") pod "53484c95-a304-4408-89e2-4ea262687889" (UID: "53484c95-a304-4408-89e2-4ea262687889"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:31:44.542483 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.542449    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-dshm" (OuterVolumeSpecName: "dshm") pod "53484c95-a304-4408-89e2-4ea262687889" (UID: "53484c95-a304-4408-89e2-4ea262687889"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:31:44.542778 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.542735    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53484c95-a304-4408-89e2-4ea262687889-kube-api-access-99fgr" (OuterVolumeSpecName: "kube-api-access-99fgr") pod "53484c95-a304-4408-89e2-4ea262687889" (UID: "53484c95-a304-4408-89e2-4ea262687889"). InnerVolumeSpecName "kube-api-access-99fgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:31:44.542778 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.542746    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53484c95-a304-4408-89e2-4ea262687889-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53484c95-a304-4408-89e2-4ea262687889" (UID: "53484c95-a304-4408-89e2-4ea262687889"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:31:44.553027 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.553000    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53484c95-a304-4408-89e2-4ea262687889" (UID: "53484c95-a304-4408-89e2-4ea262687889"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:31:44.565857 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.565812    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "53484c95-a304-4408-89e2-4ea262687889" (UID: "53484c95-a304-4408-89e2-4ea262687889"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:31:44.641511 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.641465    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:31:44.641511 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.641500    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99fgr\" (UniqueName: \"kubernetes.io/projected/53484c95-a304-4408-89e2-4ea262687889-kube-api-access-99fgr\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:31:44.641511 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.641511    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53484c95-a304-4408-89e2-4ea262687889-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:31:44.641511 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.641520    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:31:44.641876 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.641528    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:31:44.641876 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.641538    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53484c95-a304-4408-89e2-4ea262687889-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:31:44.641876 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.641552    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53484c95-a304-4408-89e2-4ea262687889-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:31:44.772892 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.772853    2587 generic.go:356] "Generic (PLEG): container finished" podID="53484c95-a304-4408-89e2-4ea262687889" containerID="ff0615c69b9ac55d22d4f0c76f2d993328aa627c818ca669fd3aab80b4138d5d" exitCode=0
Sep 17 19:31:44.773412 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.772929    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" event={"ID":"53484c95-a304-4408-89e2-4ea262687889","Type":"ContainerDied","Data":"ff0615c69b9ac55d22d4f0c76f2d993328aa627c818ca669fd3aab80b4138d5d"}
Sep 17 19:31:44.773412 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.773039    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99" event={"ID":"53484c95-a304-4408-89e2-4ea262687889","Type":"ContainerDied","Data":"35847483c00ba4c239aa06b672ce1933e879bdae467ef75cbd92dc4bfbe5e104"}
Sep 17 19:31:44.773412 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.772959    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"
Sep 17 19:31:44.773412 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.773073    2587 scope.go:117] "RemoveContainer" containerID="ff0615c69b9ac55d22d4f0c76f2d993328aa627c818ca669fd3aab80b4138d5d"
Sep 17 19:31:44.782435 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.782408    2587 scope.go:117] "RemoveContainer" containerID="cc967fbabec56c886f633ddd231d21a14e40c640741b6f7c27cc038e6a8c311c"
Sep 17 19:31:44.792634 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.792614    2587 scope.go:117] "RemoveContainer" containerID="ff0615c69b9ac55d22d4f0c76f2d993328aa627c818ca669fd3aab80b4138d5d"
Sep 17 19:31:44.793022 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:31:44.792986    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0615c69b9ac55d22d4f0c76f2d993328aa627c818ca669fd3aab80b4138d5d\": container with ID starting with ff0615c69b9ac55d22d4f0c76f2d993328aa627c818ca669fd3aab80b4138d5d not found: ID does not exist" containerID="ff0615c69b9ac55d22d4f0c76f2d993328aa627c818ca669fd3aab80b4138d5d"
Sep 17 19:31:44.793114 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.793023    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0615c69b9ac55d22d4f0c76f2d993328aa627c818ca669fd3aab80b4138d5d"} err="failed to get container status \"ff0615c69b9ac55d22d4f0c76f2d993328aa627c818ca669fd3aab80b4138d5d\": rpc error: code = NotFound desc = could not find container \"ff0615c69b9ac55d22d4f0c76f2d993328aa627c818ca669fd3aab80b4138d5d\": container with ID starting with ff0615c69b9ac55d22d4f0c76f2d993328aa627c818ca669fd3aab80b4138d5d not found: ID does not exist"
Sep 17 19:31:44.793114 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.793049    2587 scope.go:117] "RemoveContainer" containerID="cc967fbabec56c886f633ddd231d21a14e40c640741b6f7c27cc038e6a8c311c"
Sep 17 19:31:44.793369 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:31:44.793347    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc967fbabec56c886f633ddd231d21a14e40c640741b6f7c27cc038e6a8c311c\": container with ID starting with cc967fbabec56c886f633ddd231d21a14e40c640741b6f7c27cc038e6a8c311c not found: ID does not exist" containerID="cc967fbabec56c886f633ddd231d21a14e40c640741b6f7c27cc038e6a8c311c"
Sep 17 19:31:44.793428 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.793380    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc967fbabec56c886f633ddd231d21a14e40c640741b6f7c27cc038e6a8c311c"} err="failed to get container status \"cc967fbabec56c886f633ddd231d21a14e40c640741b6f7c27cc038e6a8c311c\": rpc error: code = NotFound desc = could not find container \"cc967fbabec56c886f633ddd231d21a14e40c640741b6f7c27cc038e6a8c311c\": container with ID starting with cc967fbabec56c886f633ddd231d21a14e40c640741b6f7c27cc038e6a8c311c not found: ID does not exist"
Sep 17 19:31:44.798117 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.798085    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"]
Sep 17 19:31:44.801987 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:44.801959    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-64484479fd-gxd99"]
Sep 17 19:31:46.258393 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:46.258349    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53484c95-a304-4408-89e2-4ea262687889" path="/var/lib/kubelet/pods/53484c95-a304-4408-89e2-4ea262687889/volumes"
Sep 17 19:31:48.826768 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:48.826736    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:48.930092 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:48.930056    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:49.026694 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:49.026652    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:49.128217 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:49.128124    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:49.228523 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:49.228482    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:51.586179 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:51.586147    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:51.654482 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:51.654444    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:51.788208 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:51.788160    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 19:31:52.157031 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.156990    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"]
Sep 17 19:31:52.157404 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.157386    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53484c95-a304-4408-89e2-4ea262687889" containerName="storage-initializer"
Sep 17 19:31:52.157486 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.157408    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="53484c95-a304-4408-89e2-4ea262687889" containerName="storage-initializer"
Sep 17 19:31:52.157486 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.157437    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53484c95-a304-4408-89e2-4ea262687889" containerName="main"
Sep 17 19:31:52.157486 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.157446    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="53484c95-a304-4408-89e2-4ea262687889" containerName="main"
Sep 17 19:31:52.157639 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.157528    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="53484c95-a304-4408-89e2-4ea262687889" containerName="main"
Sep 17 19:31:52.160709 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.160683    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"
Sep 17 19:31:52.163503 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.163476    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-hhcgz\""
Sep 17 19:31:52.163742 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.163725    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 19:31:52.164591 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.164577    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 19:31:52.164651 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.164628    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 19:31:52.169818 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.169788    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"]
Sep 17 19:31:52.308698 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.308661    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4mpj\" (UniqueName: \"kubernetes.io/projected/5907b341-4c48-427c-b14d-297765007ab8-kube-api-access-c4mpj\") pod \"e2e-pvc-lora-download-mlt4v\" (UID: \"5907b341-4c48-427c-b14d-297765007ab8\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"
Sep 17 19:31:52.319165 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.319137    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:52.409445 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.409345    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4mpj\" (UniqueName: \"kubernetes.io/projected/5907b341-4c48-427c-b14d-297765007ab8-kube-api-access-c4mpj\") pod \"e2e-pvc-lora-download-mlt4v\" (UID: \"5907b341-4c48-427c-b14d-297765007ab8\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"
Sep 17 19:31:52.419451 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:52.419412    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4mpj\" (UniqueName: \"kubernetes.io/projected/5907b341-4c48-427c-b14d-297765007ab8-kube-api-access-c4mpj\") pod \"e2e-pvc-lora-download-mlt4v\" (UID: \"5907b341-4c48-427c-b14d-297765007ab8\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"
Sep 17 19:31:54.731670 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:54.731631    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"e2e-pvc-lora-download-mlt4v\" (UID: \"5907b341-4c48-427c-b14d-297765007ab8\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"
Sep 17 19:31:54.737525 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:54.737499    2587 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"e2e-pvc-lora-download-mlt4v\" (UID: \"5907b341-4c48-427c-b14d-297765007ab8\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"
Sep 17 19:31:54.832129 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:54.832090    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"e2e-pvc-lora-download-mlt4v\" (UID: \"5907b341-4c48-427c-b14d-297765007ab8\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"
Sep 17 19:31:54.832311 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:54.832206    2587 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"e2e-pvc-lora-download-mlt4v\" (UID: \"5907b341-4c48-427c-b14d-297765007ab8\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"
Sep 17 19:31:54.835351 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:54.835333    2587 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"e2e-pvc-lora-download-mlt4v\" (UID: \"5907b341-4c48-427c-b14d-297765007ab8\") DevicePath \"csi-95c8e48b6ad5853f0d768fd7b3b6ce0934de2984f7ead6d618139a140631edc5\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"
Sep 17 19:31:54.987037 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:54.987003    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:55.052209 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:55.052171    2587 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"e2e-pvc-lora-download-mlt4v\" (UID: \"5907b341-4c48-427c-b14d-297765007ab8\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/321144a0c26e691fd1e7e6cae9e00cb398fbbe91f46cddd46e71e77792169862/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"
Sep 17 19:31:55.068004 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:55.067975    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"e2e-pvc-lora-download-mlt4v\" (UID: \"5907b341-4c48-427c-b14d-297765007ab8\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"
Sep 17 19:31:55.073519 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:55.073486    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:55.171098 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:55.171040    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"
Sep 17 19:31:55.447804 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:55.447620    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"]
Sep 17 19:31:55.450858 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:31:55.450830    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5907b341_4c48_427c_b14d_297765007ab8.slice/crio-a512e82a1b43511c40729d51569946848da8e60f1f35e4a465fedb8b1ecd7e0c WatchSource:0}: Error finding container a512e82a1b43511c40729d51569946848da8e60f1f35e4a465fedb8b1ecd7e0c: Status 404 returned error can't find the container with id a512e82a1b43511c40729d51569946848da8e60f1f35e4a465fedb8b1ecd7e0c
Sep 17 19:31:55.828826 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:55.828784    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v" event={"ID":"5907b341-4c48-427c-b14d-297765007ab8","Type":"ContainerStarted","Data":"133e986d0cdfb7f5d36b87b06ef780e88fb81392f0f543d487737e4c54a4c7ac"}
Sep 17 19:31:55.829276 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:55.828835    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v" event={"ID":"5907b341-4c48-427c-b14d-297765007ab8","Type":"ContainerStarted","Data":"a512e82a1b43511c40729d51569946848da8e60f1f35e4a465fedb8b1ecd7e0c"}
Sep 17 19:31:55.846073 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:55.846011    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v" podStartSLOduration=6.84598889 podStartE2EDuration="6.84598889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:31:55.844633839 +0000 UTC m=+3167.723602596" watchObservedRunningTime="2026-09-17 19:31:55.84598889 +0000 UTC m=+3167.724957647"
Sep 17 19:31:57.842089 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:57.841979    2587 generic.go:356] "Generic (PLEG): container finished" podID="5907b341-4c48-427c-b14d-297765007ab8" containerID="133e986d0cdfb7f5d36b87b06ef780e88fb81392f0f543d487737e4c54a4c7ac" exitCode=0
Sep 17 19:31:57.842089 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:57.842062    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v" event={"ID":"5907b341-4c48-427c-b14d-297765007ab8","Type":"ContainerDied","Data":"133e986d0cdfb7f5d36b87b06ef780e88fb81392f0f543d487737e4c54a4c7ac"}
Sep 17 19:31:58.990720 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:58.990684    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"
Sep 17 19:31:59.068049 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:59.068010    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"5907b341-4c48-427c-b14d-297765007ab8\" (UID: \"5907b341-4c48-427c-b14d-297765007ab8\") "
Sep 17 19:31:59.068256 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:59.068146    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4mpj\" (UniqueName: \"kubernetes.io/projected/5907b341-4c48-427c-b14d-297765007ab8-kube-api-access-c4mpj\") pod \"5907b341-4c48-427c-b14d-297765007ab8\" (UID: \"5907b341-4c48-427c-b14d-297765007ab8\") "
Sep 17 19:31:59.070639 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:59.070604    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5907b341-4c48-427c-b14d-297765007ab8-kube-api-access-c4mpj" (OuterVolumeSpecName: "kube-api-access-c4mpj") pod "5907b341-4c48-427c-b14d-297765007ab8" (UID: "5907b341-4c48-427c-b14d-297765007ab8"). InnerVolumeSpecName "kube-api-access-c4mpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:31:59.072259 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:59.072232    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75" (OuterVolumeSpecName: "model-storage") pod "5907b341-4c48-427c-b14d-297765007ab8" (UID: "5907b341-4c48-427c-b14d-297765007ab8"). InnerVolumeSpecName "pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 19:31:59.169129 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:59.169038    2587 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") on node \"ip-10-0-140-194.ec2.internal\" "
Sep 17 19:31:59.169129 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:59.169072    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4mpj\" (UniqueName: \"kubernetes.io/projected/5907b341-4c48-427c-b14d-297765007ab8-kube-api-access-c4mpj\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:31:59.854156 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:59.854125    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"
Sep 17 19:31:59.854344 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:59.854165    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v" event={"ID":"5907b341-4c48-427c-b14d-297765007ab8","Type":"ContainerDied","Data":"a512e82a1b43511c40729d51569946848da8e60f1f35e4a465fedb8b1ecd7e0c"}
Sep 17 19:31:59.854344 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:31:59.854205    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a512e82a1b43511c40729d51569946848da8e60f1f35e4a465fedb8b1ecd7e0c"
Sep 17 19:32:00.145509 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:00.145484    2587 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75") on node "ip-10-0-140-194.ec2.internal"
Sep 17 19:32:00.175761 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:00.175724    2587 reconciler_common.go:299] "Volume detached for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"csi-95c8e48b6ad5853f0d768fd7b3b6ce0934de2984f7ead6d618139a140631edc5\""
Sep 17 19:32:01.788302 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:01.788248    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 19:32:11.788374 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:11.788315    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 19:32:21.788299 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:21.788247    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 19:32:22.017149 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.017113    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"]
Sep 17 19:32:22.017476 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.017462    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5907b341-4c48-427c-b14d-297765007ab8" containerName="storage-initializer"
Sep 17 19:32:22.017528 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.017477    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5907b341-4c48-427c-b14d-297765007ab8" containerName="storage-initializer"
Sep 17 19:32:22.017564 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.017538    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="5907b341-4c48-427c-b14d-297765007ab8" containerName="storage-initializer"
Sep 17 19:32:22.022152 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.022123    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.030147 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.030103    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 19:32:22.030347 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.030149    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:32:22.030347 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.030182    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-hhcgz\""
Sep 17 19:32:22.030347 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.030194    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 17 19:32:22.030347 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.030236    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 19:32:22.030347 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.030182    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 19:32:22.037752 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.037723    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"]
Sep 17 19:32:22.047758 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.047673    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:22.173885 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.173842    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e660cbf-c7ce-4911-b0fb-290708930aa1-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.173885 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.173893    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.174150 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.173931    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-home\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.174150 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.174009    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.174150 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.174052    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.174150 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.174078    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e660cbf-c7ce-4911-b0fb-290708930aa1-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.174150 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.174121    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwrz6\" (UniqueName: \"kubernetes.io/projected/9e660cbf-c7ce-4911-b0fb-290708930aa1-kube-api-access-wwrz6\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.174314 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.174155    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-dshm\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.274905 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.274863    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.275132 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.274963    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.275132 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.274995    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e660cbf-c7ce-4911-b0fb-290708930aa1-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.275132 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.275030    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwrz6\" (UniqueName: \"kubernetes.io/projected/9e660cbf-c7ce-4911-b0fb-290708930aa1-kube-api-access-wwrz6\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.275132 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.275082    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-dshm\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.275132 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.275111    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e660cbf-c7ce-4911-b0fb-290708930aa1-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.275393 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.275139    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.275393 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.275179    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-home\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.275393 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.275385    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.275682 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.275655    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.275765 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.275693    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.275765 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.275725    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-home\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.276604 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.276578    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e660cbf-c7ce-4911-b0fb-290708930aa1-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.277709 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.277683    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-dshm\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.278010 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.277990    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e660cbf-c7ce-4911-b0fb-290708930aa1-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:22.284778 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:22.284748    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwrz6\" (UniqueName: \"kubernetes.io/projected/9e660cbf-c7ce-4911-b0fb-290708930aa1-kube-api-access-wwrz6\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:25.404876 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:25.404833    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:25.410880 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:25.410853    2587 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:25.505825 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:25.505787    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:25.506064 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:25.505963    2587 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:25.509023 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:25.509001    2587 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") DevicePath \"csi-95c8e48b6ad5853f0d768fd7b3b6ce0934de2984f7ead6d618139a140631edc5\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:25.599633 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:25.599587    2587 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/321144a0c26e691fd1e7e6cae9e00cb398fbbe91f46cddd46e71e77792169862/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:25.621130 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:25.621094    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:25.641627 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:25.641583    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:25.950108 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:25.950078    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"]
Sep 17 19:32:25.952576 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:32:25.952543    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e660cbf_c7ce_4911_b0fb_290708930aa1.slice/crio-7bcfd8f22588063334388c0743ee4f9afc1c80a2a7314495eecccedb0bb6ed69 WatchSource:0}: Error finding container 7bcfd8f22588063334388c0743ee4f9afc1c80a2a7314495eecccedb0bb6ed69: Status 404 returned error can't find the container with id 7bcfd8f22588063334388c0743ee4f9afc1c80a2a7314495eecccedb0bb6ed69
Sep 17 19:32:25.978810 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:25.978769    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" event={"ID":"9e660cbf-c7ce-4911-b0fb-290708930aa1","Type":"ContainerStarted","Data":"7bcfd8f22588063334388c0743ee4f9afc1c80a2a7314495eecccedb0bb6ed69"}
Sep 17 19:32:26.986846 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:26.986805    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" event={"ID":"9e660cbf-c7ce-4911-b0fb-290708930aa1","Type":"ContainerStarted","Data":"c0a0ff9d32d4d555222ed3dbac64eb9dad4796727ed429a611cbe30cb8e1ae7b"}
Sep 17 19:32:28.999658 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:28.999619    2587 generic.go:356] "Generic (PLEG): container finished" podID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerID="c0a0ff9d32d4d555222ed3dbac64eb9dad4796727ed429a611cbe30cb8e1ae7b" exitCode=0
Sep 17 19:32:29.000099 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:28.999700    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" event={"ID":"9e660cbf-c7ce-4911-b0fb-290708930aa1","Type":"ContainerDied","Data":"c0a0ff9d32d4d555222ed3dbac64eb9dad4796727ed429a611cbe30cb8e1ae7b"}
Sep 17 19:32:30.008206 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:30.008169    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" event={"ID":"9e660cbf-c7ce-4911-b0fb-290708930aa1","Type":"ContainerStarted","Data":"8ecaddc149387c0ebf98c87f30d6550be0f6f1532d537d6786c56bb27c49a9ed"}
Sep 17 19:32:30.032130 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:30.032064    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" podStartSLOduration=6.604421169 podStartE2EDuration="9.032041894s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.427620725s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:32:30.031053953 +0000 UTC m=+3201.910022688" watchObservedRunningTime="2026-09-17 19:32:30.032041894 +0000 UTC m=+3201.911010655"
Sep 17 19:32:30.642132 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:30.642085    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:31.788730 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:31.788686    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 19:32:35.642493 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:35.642433    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:32:35.645078 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:35.645045    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 19:32:41.788770 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:41.788721    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 19:32:45.642691 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:45.642636    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 19:32:51.788182 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:51.788073    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 19:32:55.642340 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:32:55.642286    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 19:33:01.789055 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:01.788994    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 19:33:05.642036 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:05.641994    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 19:33:11.788592 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:11.788541    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 17 19:33:15.642764 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:15.642714    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 19:33:21.798225 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:21.798186    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:33:21.806136 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:21.806104    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:33:25.642556 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:25.642505    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 19:33:30.963268 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:30.963237    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:30.999541 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:30.999502    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"]
Sep 17 19:33:31.049393 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:31.049361    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:35.642460 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:35.642413    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 19:33:45.652854 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:45.652818    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:33:45.660797 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:45.660773    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:33:46.023009 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.022960    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main" containerID="cri-o://224630598a06224d811a825d39dee4fca6670af636a0a3b2e80a9973ce75874d" gracePeriod=45
Sep 17 19:33:46.778012 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.777983    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:33:46.878241 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.878152    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-model-cache\") pod \"22462499-1df6-4fde-aec4-69b1657aed2e\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") "
Sep 17 19:33:46.878241 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.878193    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-home\") pod \"22462499-1df6-4fde-aec4-69b1657aed2e\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") "
Sep 17 19:33:46.878241 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.878232    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22462499-1df6-4fde-aec4-69b1657aed2e-tls-certs\") pod \"22462499-1df6-4fde-aec4-69b1657aed2e\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") "
Sep 17 19:33:46.878548 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.878262    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-dshm\") pod \"22462499-1df6-4fde-aec4-69b1657aed2e\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") "
Sep 17 19:33:46.878548 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.878284    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22462499-1df6-4fde-aec4-69b1657aed2e-cabundle-cert\") pod \"22462499-1df6-4fde-aec4-69b1657aed2e\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") "
Sep 17 19:33:46.878548 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.878318    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-kserve-provision-location\") pod \"22462499-1df6-4fde-aec4-69b1657aed2e\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") "
Sep 17 19:33:46.878548 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.878340    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn8fs\" (UniqueName: \"kubernetes.io/projected/22462499-1df6-4fde-aec4-69b1657aed2e-kube-api-access-nn8fs\") pod \"22462499-1df6-4fde-aec4-69b1657aed2e\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") "
Sep 17 19:33:46.878548 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.878376    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-tmp-dir\") pod \"22462499-1df6-4fde-aec4-69b1657aed2e\" (UID: \"22462499-1df6-4fde-aec4-69b1657aed2e\") "
Sep 17 19:33:46.881218 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.878849    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22462499-1df6-4fde-aec4-69b1657aed2e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22462499-1df6-4fde-aec4-69b1657aed2e" (UID: "22462499-1df6-4fde-aec4-69b1657aed2e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:33:46.881218 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.878991    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-home" (OuterVolumeSpecName: "home") pod "22462499-1df6-4fde-aec4-69b1657aed2e" (UID: "22462499-1df6-4fde-aec4-69b1657aed2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:46.881218 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.879122    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-model-cache" (OuterVolumeSpecName: "model-cache") pod "22462499-1df6-4fde-aec4-69b1657aed2e" (UID: "22462499-1df6-4fde-aec4-69b1657aed2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:46.881473 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.881286    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-dshm" (OuterVolumeSpecName: "dshm") pod "22462499-1df6-4fde-aec4-69b1657aed2e" (UID: "22462499-1df6-4fde-aec4-69b1657aed2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:46.881473 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.881303    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22462499-1df6-4fde-aec4-69b1657aed2e-kube-api-access-nn8fs" (OuterVolumeSpecName: "kube-api-access-nn8fs") pod "22462499-1df6-4fde-aec4-69b1657aed2e" (UID: "22462499-1df6-4fde-aec4-69b1657aed2e"). InnerVolumeSpecName "kube-api-access-nn8fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:33:46.881981 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.881934    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22462499-1df6-4fde-aec4-69b1657aed2e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22462499-1df6-4fde-aec4-69b1657aed2e" (UID: "22462499-1df6-4fde-aec4-69b1657aed2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:33:46.897263 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.897221    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22462499-1df6-4fde-aec4-69b1657aed2e" (UID: "22462499-1df6-4fde-aec4-69b1657aed2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:46.906131 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.906099    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22462499-1df6-4fde-aec4-69b1657aed2e" (UID: "22462499-1df6-4fde-aec4-69b1657aed2e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:46.980323 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.980276    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:33:46.980323 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.980312    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:33:46.980323 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.980324    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:33:46.980323 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.980334    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22462499-1df6-4fde-aec4-69b1657aed2e-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:33:46.980612 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.980344    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:33:46.980612 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.980356    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22462499-1df6-4fde-aec4-69b1657aed2e-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:33:46.980612 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.980368    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22462499-1df6-4fde-aec4-69b1657aed2e-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:33:46.980612 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:46.980379    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn8fs\" (UniqueName: \"kubernetes.io/projected/22462499-1df6-4fde-aec4-69b1657aed2e-kube-api-access-nn8fs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:33:47.416423 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:47.416387    2587 generic.go:356] "Generic (PLEG): container finished" podID="22462499-1df6-4fde-aec4-69b1657aed2e" containerID="224630598a06224d811a825d39dee4fca6670af636a0a3b2e80a9973ce75874d" exitCode=0
Sep 17 19:33:47.416596 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:47.416445    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"
Sep 17 19:33:47.416596 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:47.416478    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" event={"ID":"22462499-1df6-4fde-aec4-69b1657aed2e","Type":"ContainerDied","Data":"224630598a06224d811a825d39dee4fca6670af636a0a3b2e80a9973ce75874d"}
Sep 17 19:33:47.416596 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:47.416540    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd" event={"ID":"22462499-1df6-4fde-aec4-69b1657aed2e","Type":"ContainerDied","Data":"ad7079e8fc265f50cf6f72b8327a4dc55e1caeaa259182672c6077514931a89f"}
Sep 17 19:33:47.416596 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:47.416571    2587 scope.go:117] "RemoveContainer" containerID="224630598a06224d811a825d39dee4fca6670af636a0a3b2e80a9973ce75874d"
Sep 17 19:33:47.425609 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:47.425585    2587 scope.go:117] "RemoveContainer" containerID="d34d344c8e34563bdf27c3827c12b1557798843da88016717e281db0f836c061"
Sep 17 19:33:47.433818 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:47.433795    2587 scope.go:117] "RemoveContainer" containerID="224630598a06224d811a825d39dee4fca6670af636a0a3b2e80a9973ce75874d"
Sep 17 19:33:47.434117 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:33:47.434097    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224630598a06224d811a825d39dee4fca6670af636a0a3b2e80a9973ce75874d\": container with ID starting with 224630598a06224d811a825d39dee4fca6670af636a0a3b2e80a9973ce75874d not found: ID does not exist" containerID="224630598a06224d811a825d39dee4fca6670af636a0a3b2e80a9973ce75874d"
Sep 17 19:33:47.434176 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:47.434125    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224630598a06224d811a825d39dee4fca6670af636a0a3b2e80a9973ce75874d"} err="failed to get container status \"224630598a06224d811a825d39dee4fca6670af636a0a3b2e80a9973ce75874d\": rpc error: code = NotFound desc = could not find container \"224630598a06224d811a825d39dee4fca6670af636a0a3b2e80a9973ce75874d\": container with ID starting with 224630598a06224d811a825d39dee4fca6670af636a0a3b2e80a9973ce75874d not found: ID does not exist"
Sep 17 19:33:47.434176 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:47.434145    2587 scope.go:117] "RemoveContainer" containerID="d34d344c8e34563bdf27c3827c12b1557798843da88016717e281db0f836c061"
Sep 17 19:33:47.434383 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:33:47.434368    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34d344c8e34563bdf27c3827c12b1557798843da88016717e281db0f836c061\": container with ID starting with d34d344c8e34563bdf27c3827c12b1557798843da88016717e281db0f836c061 not found: ID does not exist" containerID="d34d344c8e34563bdf27c3827c12b1557798843da88016717e281db0f836c061"
Sep 17 19:33:47.434422 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:47.434389    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34d344c8e34563bdf27c3827c12b1557798843da88016717e281db0f836c061"} err="failed to get container status \"d34d344c8e34563bdf27c3827c12b1557798843da88016717e281db0f836c061\": rpc error: code = NotFound desc = could not find container \"d34d344c8e34563bdf27c3827c12b1557798843da88016717e281db0f836c061\": container with ID starting with d34d344c8e34563bdf27c3827c12b1557798843da88016717e281db0f836c061 not found: ID does not exist"
Sep 17 19:33:47.444084 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:47.444056    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"]
Sep 17 19:33:47.447572 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:47.447544    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5d6497cbb4-7jnvd"]
Sep 17 19:33:48.257880 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:33:48.257840    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" path="/var/lib/kubelet/pods/22462499-1df6-4fde-aec4-69b1657aed2e/volumes"
Sep 17 19:34:10.436397 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:10.436354    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:34:10.438656 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:10.438628    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:34:11.309741 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:11.309701    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs_9e660cbf-c7ce-4911-b0fb-290708930aa1/storage-initializer/0.log"
Sep 17 19:34:11.318784 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:11.318757    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs_9e660cbf-c7ce-4911-b0fb-290708930aa1/main/0.log"
Sep 17 19:34:12.413235 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:12.413192    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"]
Sep 17 19:34:12.420674 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:12.420640    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:13.300748 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:13.300701    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:21.124492 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:21.124386    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:21.222803 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:21.222770    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:27.188721 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:27.188685    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:27.442316 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:27.442227    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerName="main" containerID="cri-o://8ecaddc149387c0ebf98c87f30d6550be0f6f1532d537d6786c56bb27c49a9ed" gracePeriod=45
Sep 17 19:34:27.654039 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:27.653999    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 17 19:34:28.096934 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.096903    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:34:28.247341 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.247315    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-tmp-dir\") pod \"9e660cbf-c7ce-4911-b0fb-290708930aa1\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") "
Sep 17 19:34:28.247838 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.247349    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-dshm\") pod \"9e660cbf-c7ce-4911-b0fb-290708930aa1\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") "
Sep 17 19:34:28.247838 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.247447    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") pod \"9e660cbf-c7ce-4911-b0fb-290708930aa1\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") "
Sep 17 19:34:28.247838 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.247475    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-home\") pod \"9e660cbf-c7ce-4911-b0fb-290708930aa1\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") "
Sep 17 19:34:28.247838 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.247495    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e660cbf-c7ce-4911-b0fb-290708930aa1-cabundle-cert\") pod \"9e660cbf-c7ce-4911-b0fb-290708930aa1\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") "
Sep 17 19:34:28.247838 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.247510    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-model-cache\") pod \"9e660cbf-c7ce-4911-b0fb-290708930aa1\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") "
Sep 17 19:34:28.247838 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.247544    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e660cbf-c7ce-4911-b0fb-290708930aa1-tls-certs\") pod \"9e660cbf-c7ce-4911-b0fb-290708930aa1\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") "
Sep 17 19:34:28.247838 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.247613    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-kserve-provision-location\") pod \"9e660cbf-c7ce-4911-b0fb-290708930aa1\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") "
Sep 17 19:34:28.247838 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.247644    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwrz6\" (UniqueName: \"kubernetes.io/projected/9e660cbf-c7ce-4911-b0fb-290708930aa1-kube-api-access-wwrz6\") pod \"9e660cbf-c7ce-4911-b0fb-290708930aa1\" (UID: \"9e660cbf-c7ce-4911-b0fb-290708930aa1\") "
Sep 17 19:34:28.248309 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.247868    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e660cbf-c7ce-4911-b0fb-290708930aa1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9e660cbf-c7ce-4911-b0fb-290708930aa1" (UID: "9e660cbf-c7ce-4911-b0fb-290708930aa1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:34:28.248309 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.247899    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-home" (OuterVolumeSpecName: "home") pod "9e660cbf-c7ce-4911-b0fb-290708930aa1" (UID: "9e660cbf-c7ce-4911-b0fb-290708930aa1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:28.248309 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.248124    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e660cbf-c7ce-4911-b0fb-290708930aa1" (UID: "9e660cbf-c7ce-4911-b0fb-290708930aa1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:28.250423 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.250395    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-dshm" (OuterVolumeSpecName: "dshm") pod "9e660cbf-c7ce-4911-b0fb-290708930aa1" (UID: "9e660cbf-c7ce-4911-b0fb-290708930aa1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:28.250571 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.250447    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e660cbf-c7ce-4911-b0fb-290708930aa1-kube-api-access-wwrz6" (OuterVolumeSpecName: "kube-api-access-wwrz6") pod "9e660cbf-c7ce-4911-b0fb-290708930aa1" (UID: "9e660cbf-c7ce-4911-b0fb-290708930aa1"). InnerVolumeSpecName "kube-api-access-wwrz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:34:28.250787 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.250764    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e660cbf-c7ce-4911-b0fb-290708930aa1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e660cbf-c7ce-4911-b0fb-290708930aa1" (UID: "9e660cbf-c7ce-4911-b0fb-290708930aa1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:34:28.253417 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.253389    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "9e660cbf-c7ce-4911-b0fb-290708930aa1" (UID: "9e660cbf-c7ce-4911-b0fb-290708930aa1"). InnerVolumeSpecName "pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 19:34:28.267955 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.267904    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e660cbf-c7ce-4911-b0fb-290708930aa1" (UID: "9e660cbf-c7ce-4911-b0fb-290708930aa1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:28.279770 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.279733    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9e660cbf-c7ce-4911-b0fb-290708930aa1" (UID: "9e660cbf-c7ce-4911-b0fb-290708930aa1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:28.348696 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.348652    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:34:28.348884 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.348707    2587 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") on node \"ip-10-0-140-194.ec2.internal\" "
Sep 17 19:34:28.348884 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.348722    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:34:28.348884 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.348731    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e660cbf-c7ce-4911-b0fb-290708930aa1-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:34:28.348884 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.348740    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:34:28.348884 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.348747    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e660cbf-c7ce-4911-b0fb-290708930aa1-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:34:28.348884 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.348755    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:34:28.348884 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.348766    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwrz6\" (UniqueName: \"kubernetes.io/projected/9e660cbf-c7ce-4911-b0fb-290708930aa1-kube-api-access-wwrz6\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:34:28.348884 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.348779    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e660cbf-c7ce-4911-b0fb-290708930aa1-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:34:28.358558 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.358537    2587 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75") on node "ip-10-0-140-194.ec2.internal"
Sep 17 19:34:28.449957 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.449899    2587 reconciler_common.go:299] "Volume detached for volume \"pvc-5c685465-e6b5-4990-902c-6c0d4f8eb307\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0504e968367212b75\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"csi-95c8e48b6ad5853f0d768fd7b3b6ce0934de2984f7ead6d618139a140631edc5\""
Sep 17 19:34:28.615620 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.615532    2587 generic.go:356] "Generic (PLEG): container finished" podID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerID="8ecaddc149387c0ebf98c87f30d6550be0f6f1532d537d6786c56bb27c49a9ed" exitCode=0
Sep 17 19:34:28.615620 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.615589    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"
Sep 17 19:34:28.615862 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.615626    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" event={"ID":"9e660cbf-c7ce-4911-b0fb-290708930aa1","Type":"ContainerDied","Data":"8ecaddc149387c0ebf98c87f30d6550be0f6f1532d537d6786c56bb27c49a9ed"}
Sep 17 19:34:28.615862 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.615693    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs" event={"ID":"9e660cbf-c7ce-4911-b0fb-290708930aa1","Type":"ContainerDied","Data":"7bcfd8f22588063334388c0743ee4f9afc1c80a2a7314495eecccedb0bb6ed69"}
Sep 17 19:34:28.615862 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.615723    2587 scope.go:117] "RemoveContainer" containerID="8ecaddc149387c0ebf98c87f30d6550be0f6f1532d537d6786c56bb27c49a9ed"
Sep 17 19:34:28.624829 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.624813    2587 scope.go:117] "RemoveContainer" containerID="c0a0ff9d32d4d555222ed3dbac64eb9dad4796727ed429a611cbe30cb8e1ae7b"
Sep 17 19:34:28.632403 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.632381    2587 scope.go:117] "RemoveContainer" containerID="8ecaddc149387c0ebf98c87f30d6550be0f6f1532d537d6786c56bb27c49a9ed"
Sep 17 19:34:28.632664 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:34:28.632645    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ecaddc149387c0ebf98c87f30d6550be0f6f1532d537d6786c56bb27c49a9ed\": container with ID starting with 8ecaddc149387c0ebf98c87f30d6550be0f6f1532d537d6786c56bb27c49a9ed not found: ID does not exist" containerID="8ecaddc149387c0ebf98c87f30d6550be0f6f1532d537d6786c56bb27c49a9ed"
Sep 17 19:34:28.632720 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.632673    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecaddc149387c0ebf98c87f30d6550be0f6f1532d537d6786c56bb27c49a9ed"} err="failed to get container status \"8ecaddc149387c0ebf98c87f30d6550be0f6f1532d537d6786c56bb27c49a9ed\": rpc error: code = NotFound desc = could not find container \"8ecaddc149387c0ebf98c87f30d6550be0f6f1532d537d6786c56bb27c49a9ed\": container with ID starting with 8ecaddc149387c0ebf98c87f30d6550be0f6f1532d537d6786c56bb27c49a9ed not found: ID does not exist"
Sep 17 19:34:28.632720 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.632692    2587 scope.go:117] "RemoveContainer" containerID="c0a0ff9d32d4d555222ed3dbac64eb9dad4796727ed429a611cbe30cb8e1ae7b"
Sep 17 19:34:28.632983 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:34:28.632959    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a0ff9d32d4d555222ed3dbac64eb9dad4796727ed429a611cbe30cb8e1ae7b\": container with ID starting with c0a0ff9d32d4d555222ed3dbac64eb9dad4796727ed429a611cbe30cb8e1ae7b not found: ID does not exist" containerID="c0a0ff9d32d4d555222ed3dbac64eb9dad4796727ed429a611cbe30cb8e1ae7b"
Sep 17 19:34:28.633061 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.632994    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a0ff9d32d4d555222ed3dbac64eb9dad4796727ed429a611cbe30cb8e1ae7b"} err="failed to get container status \"c0a0ff9d32d4d555222ed3dbac64eb9dad4796727ed429a611cbe30cb8e1ae7b\": rpc error: code = NotFound desc = could not find container \"c0a0ff9d32d4d555222ed3dbac64eb9dad4796727ed429a611cbe30cb8e1ae7b\": container with ID starting with c0a0ff9d32d4d555222ed3dbac64eb9dad4796727ed429a611cbe30cb8e1ae7b not found: ID does not exist"
Sep 17 19:34:28.639670 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.639645    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"]
Sep 17 19:34:28.643895 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:28.643872    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66ddf56f7-85jfs"]
Sep 17 19:34:30.258083 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:30.258052    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" path="/var/lib/kubelet/pods/9e660cbf-c7ce-4911-b0fb-290708930aa1/volumes"
Sep 17 19:34:33.015553 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:33.015517    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:33.121434 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:33.121395    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:33.218797 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:33.218764    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:33.321253 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:33.321175    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:33.418480 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:33.418451    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:35.772377 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:35.772344    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:35.841807 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:35.841778    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:36.334998 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.334957    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"]
Sep 17 19:34:36.335288 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.335275    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerName="main"
Sep 17 19:34:36.335339 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.335290    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerName="main"
Sep 17 19:34:36.335339 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.335307    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="storage-initializer"
Sep 17 19:34:36.335339 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.335312    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="storage-initializer"
Sep 17 19:34:36.335339 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.335321    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerName="storage-initializer"
Sep 17 19:34:36.335339 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.335327    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerName="storage-initializer"
Sep 17 19:34:36.335563 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.335342    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main"
Sep 17 19:34:36.335563 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.335347    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main"
Sep 17 19:34:36.335563 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.335395    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="22462499-1df6-4fde-aec4-69b1657aed2e" containerName="main"
Sep 17 19:34:36.335563 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.335404    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e660cbf-c7ce-4911-b0fb-290708930aa1" containerName="main"
Sep 17 19:34:36.339780 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.339758    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"
Sep 17 19:34:36.342581 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.342555    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 19:34:36.342691 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.342584    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 17 19:34:36.343832 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.343808    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 19:34:36.343986 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.343808    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-k7j6h\""
Sep 17 19:34:36.345599 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.345575    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"]
Sep 17 19:34:36.484814 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.484784    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:36.509876 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.509840    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw67x\" (UniqueName: \"kubernetes.io/projected/99937b63-2642-4100-beec-008ca8776d30-kube-api-access-xw67x\") pod \"e2e-pvc-lora-download-nsgfv\" (UID: \"99937b63-2642-4100-beec-008ca8776d30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"
Sep 17 19:34:36.610688 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.610600    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw67x\" (UniqueName: \"kubernetes.io/projected/99937b63-2642-4100-beec-008ca8776d30-kube-api-access-xw67x\") pod \"e2e-pvc-lora-download-nsgfv\" (UID: \"99937b63-2642-4100-beec-008ca8776d30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"
Sep 17 19:34:36.620456 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:36.620430    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw67x\" (UniqueName: \"kubernetes.io/projected/99937b63-2642-4100-beec-008ca8776d30-kube-api-access-xw67x\") pod \"e2e-pvc-lora-download-nsgfv\" (UID: \"99937b63-2642-4100-beec-008ca8776d30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"
Sep 17 19:34:37.998832 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:37.998790    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"]
Sep 17 19:34:38.001459 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:38.001432    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mlt4v"]
Sep 17 19:34:38.257556 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:38.257521    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5907b341-4c48-427c-b14d-297765007ab8" path="/var/lib/kubelet/pods/5907b341-4c48-427c-b14d-297765007ab8/volumes"
Sep 17 19:34:38.827850 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:38.827801    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"e2e-pvc-lora-download-nsgfv\" (UID: \"99937b63-2642-4100-beec-008ca8776d30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"
Sep 17 19:34:38.834245 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:38.834205    2587 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"e2e-pvc-lora-download-nsgfv\" (UID: \"99937b63-2642-4100-beec-008ca8776d30\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"
Sep 17 19:34:38.929290 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:38.929250    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"e2e-pvc-lora-download-nsgfv\" (UID: \"99937b63-2642-4100-beec-008ca8776d30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"
Sep 17 19:34:38.929468 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:38.929397    2587 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"e2e-pvc-lora-download-nsgfv\" (UID: \"99937b63-2642-4100-beec-008ca8776d30\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"
Sep 17 19:34:38.932425 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:38.932407    2587 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"e2e-pvc-lora-download-nsgfv\" (UID: \"99937b63-2642-4100-beec-008ca8776d30\") DevicePath \"csi-2b5ecf5b09b653f8b4ebe36a4ff39316edfcd18ef12399f2ed871a24fc964651\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"
Sep 17 19:34:38.993867 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:38.993829    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:39.093891 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:39.093806    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:39.195384 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:39.195325    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:39.236646 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:39.236614    2587 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"e2e-pvc-lora-download-nsgfv\" (UID: \"99937b63-2642-4100-beec-008ca8776d30\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/e7320d8a547ca060c7b87daa9748e7e7439b749410ef06b584b42b4d24b8df4c/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"
Sep 17 19:34:39.250705 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:39.250675    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"e2e-pvc-lora-download-nsgfv\" (UID: \"99937b63-2642-4100-beec-008ca8776d30\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"
Sep 17 19:34:39.349842 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:39.349746    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"
Sep 17 19:34:39.622052 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:39.622024    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"]
Sep 17 19:34:39.624424 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:34:39.624395    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99937b63_2642_4100_beec_008ca8776d30.slice/crio-a54a89aff75de1956081a649365373971adafe24fd71be9945ba1e8a9f372b54 WatchSource:0}: Error finding container a54a89aff75de1956081a649365373971adafe24fd71be9945ba1e8a9f372b54: Status 404 returned error can't find the container with id a54a89aff75de1956081a649365373971adafe24fd71be9945ba1e8a9f372b54
Sep 17 19:34:39.679758 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:39.679722    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv" event={"ID":"99937b63-2642-4100-beec-008ca8776d30","Type":"ContainerStarted","Data":"a54a89aff75de1956081a649365373971adafe24fd71be9945ba1e8a9f372b54"}
Sep 17 19:34:40.688161 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:40.688118    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv" event={"ID":"99937b63-2642-4100-beec-008ca8776d30","Type":"ContainerStarted","Data":"fc1b351ff5dd83fb4eb5d012b9dfc5d62a8afa7a10334e56d429eaab10b69ba4"}
Sep 17 19:34:40.705135 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:40.705084    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv" podStartSLOduration=7.705068276 podStartE2EDuration="7.705068276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:34:40.704818673 +0000 UTC m=+3332.583787431" watchObservedRunningTime="2026-09-17 19:34:40.705068276 +0000 UTC m=+3332.584037059"
Sep 17 19:34:41.463825 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:41.463788    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:42.698962 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:42.698905    2587 generic.go:356] "Generic (PLEG): container finished" podID="99937b63-2642-4100-beec-008ca8776d30" containerID="fc1b351ff5dd83fb4eb5d012b9dfc5d62a8afa7a10334e56d429eaab10b69ba4" exitCode=0
Sep 17 19:34:42.699374 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:42.698979    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv" event={"ID":"99937b63-2642-4100-beec-008ca8776d30","Type":"ContainerDied","Data":"fc1b351ff5dd83fb4eb5d012b9dfc5d62a8afa7a10334e56d429eaab10b69ba4"}
Sep 17 19:34:43.844328 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:43.844294    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"
Sep 17 19:34:43.976899 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:43.976804    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xw67x\" (UniqueName: \"kubernetes.io/projected/99937b63-2642-4100-beec-008ca8776d30-kube-api-access-xw67x\") pod \"99937b63-2642-4100-beec-008ca8776d30\" (UID: \"99937b63-2642-4100-beec-008ca8776d30\") "
Sep 17 19:34:43.977090 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:43.976979    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") pod \"99937b63-2642-4100-beec-008ca8776d30\" (UID: \"99937b63-2642-4100-beec-008ca8776d30\") "
Sep 17 19:34:43.979224 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:43.979197    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99937b63-2642-4100-beec-008ca8776d30-kube-api-access-xw67x" (OuterVolumeSpecName: "kube-api-access-xw67x") pod "99937b63-2642-4100-beec-008ca8776d30" (UID: "99937b63-2642-4100-beec-008ca8776d30"). InnerVolumeSpecName "kube-api-access-xw67x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:34:43.979812 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:43.979789    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef" (OuterVolumeSpecName: "model-storage") pod "99937b63-2642-4100-beec-008ca8776d30" (UID: "99937b63-2642-4100-beec-008ca8776d30"). InnerVolumeSpecName "pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 19:34:44.077512 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:44.077473    2587 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") on node \"ip-10-0-140-194.ec2.internal\" "
Sep 17 19:34:44.077512 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:44.077509    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw67x\" (UniqueName: \"kubernetes.io/projected/99937b63-2642-4100-beec-008ca8776d30-kube-api-access-xw67x\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:34:44.709082 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:44.708990    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv" event={"ID":"99937b63-2642-4100-beec-008ca8776d30","Type":"ContainerDied","Data":"a54a89aff75de1956081a649365373971adafe24fd71be9945ba1e8a9f372b54"}
Sep 17 19:34:44.709082 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:44.709034    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a54a89aff75de1956081a649365373971adafe24fd71be9945ba1e8a9f372b54"
Sep 17 19:34:44.709082 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:44.709004    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"
Sep 17 19:34:45.051370 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:45.051348    2587 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef") on node "ip-10-0-140-194.ec2.internal"
Sep 17 19:34:45.086400 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:45.086366    2587 reconciler_common.go:299] "Volume detached for volume \"pvc-4b0c0ad6-bb52-4dfb-98ba-67b10c6b2c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fc5e4c501c65f0ef\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"csi-2b5ecf5b09b653f8b4ebe36a4ff39316edfcd18ef12399f2ed871a24fc964651\""
Sep 17 19:34:45.119165 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:45.119113    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:45.215821 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:45.215768    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:46.155813 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:46.155771    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:46.262030 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:46.257458    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:47.272428 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:47.272396    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:48.417110 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:48.417072    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:54.306836 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:54.306796    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:56.723211 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:34:56.723171    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:13.918403 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:35:13.918371    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:17.724855 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:35:17.724808    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:17.787725 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:35:17.787691    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:40.661062 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:35:40.661028    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:40.730031 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:35:40.730006    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:06.731050 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:06.731018    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:19.120717 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:19.120682    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:21.926467 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:21.926385    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:26.905439 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:26.905378    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:27.000507 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:27.000475    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:33.238182 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:33.238133    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:38.961399 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:38.961357    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:39.118718 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:39.118678    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:39.738298 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:39.738265    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:39.896340 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:39.896308    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:40.141217 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.141132    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:40.395283 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.395193    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:40.495379 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.495344    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:40.588240 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.588202    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 19:37:40.588538 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.588524    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99937b63-2642-4100-beec-008ca8776d30" containerName="storage-initializer"
Sep 17 19:37:40.588585 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.588541    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="99937b63-2642-4100-beec-008ca8776d30" containerName="storage-initializer"
Sep 17 19:37:40.588620 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.588609    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="99937b63-2642-4100-beec-008ca8776d30" containerName="storage-initializer"
Sep 17 19:37:40.591729 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.591709    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.595406 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.595370    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:37:40.595592 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.595411    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-qbhvm\""
Sep 17 19:37:40.595676 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.595645    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 19:37:40.595724 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.595667    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 19:37:40.596364 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.596346    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 19:37:40.596364 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.596356    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-c2th7\""
Sep 17 19:37:40.596569 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.596501    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 19:37:40.617794 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.617752    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 19:37:40.703858 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.703762    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b874cca-d362-4260-980e-24e4b88c1140-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.703858 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.703808    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.703858 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.703841    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.704131 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.703885    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmnvh\" (UniqueName: \"kubernetes.io/projected/1b874cca-d362-4260-980e-24e4b88c1140-kube-api-access-dmnvh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.704131 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.703934    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.704131 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.704001    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b874cca-d362-4260-980e-24e4b88c1140-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.704131 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.704054    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.704131 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.704076    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.786198 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.786156    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"]
Sep 17 19:37:40.789511 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.789489    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"
Sep 17 19:37:40.792315 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.792289    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-9c2fl\""
Sep 17 19:37:40.803236 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.803197    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"]
Sep 17 19:37:40.804702 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.804676    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.804841 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.804715    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmnvh\" (UniqueName: \"kubernetes.io/projected/1b874cca-d362-4260-980e-24e4b88c1140-kube-api-access-dmnvh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.804906 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.804841    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.804906 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.804891    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b874cca-d362-4260-980e-24e4b88c1140-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.805060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.804922    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.805060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.804960    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.805060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.805010    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b874cca-d362-4260-980e-24e4b88c1140-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.805060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.805043    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.805263 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.805041    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.805263 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.805225    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.805382 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.805358    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.805435 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.805387    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.805629 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.805607    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b874cca-d362-4260-980e-24e4b88c1140-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.807524 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.807493    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.807965 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.807924    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b874cca-d362-4260-980e-24e4b88c1140-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.817050 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.817021    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmnvh\" (UniqueName: \"kubernetes.io/projected/1b874cca-d362-4260-980e-24e4b88c1140-kube-api-access-dmnvh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.901718 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.901682    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:37:40.905648 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.905615    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnf6g\" (UniqueName: \"kubernetes.io/projected/f1400cba-fe90-4ff1-93fa-f90c315f721f-kube-api-access-hnf6g\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8\" (UID: \"f1400cba-fe90-4ff1-93fa-f90c315f721f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"
Sep 17 19:37:40.906148 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:40.905683    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1400cba-fe90-4ff1-93fa-f90c315f721f-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8\" (UID: \"f1400cba-fe90-4ff1-93fa-f90c315f721f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"
Sep 17 19:37:41.006390 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:41.006359    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1400cba-fe90-4ff1-93fa-f90c315f721f-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8\" (UID: \"f1400cba-fe90-4ff1-93fa-f90c315f721f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"
Sep 17 19:37:41.006574 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:41.006463    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnf6g\" (UniqueName: \"kubernetes.io/projected/f1400cba-fe90-4ff1-93fa-f90c315f721f-kube-api-access-hnf6g\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8\" (UID: \"f1400cba-fe90-4ff1-93fa-f90c315f721f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"
Sep 17 19:37:41.009137 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:41.009113    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1400cba-fe90-4ff1-93fa-f90c315f721f-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8\" (UID: \"f1400cba-fe90-4ff1-93fa-f90c315f721f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"
Sep 17 19:37:41.015764 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:41.015733    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnf6g\" (UniqueName: \"kubernetes.io/projected/f1400cba-fe90-4ff1-93fa-f90c315f721f-kube-api-access-hnf6g\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8\" (UID: \"f1400cba-fe90-4ff1-93fa-f90c315f721f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"
Sep 17 19:37:41.064689 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:41.064654    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:41.099356 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:41.099312    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"
Sep 17 19:37:41.196719 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:41.196604    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 19:37:41.199517 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:37:41.199479    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b874cca_d362_4260_980e_24e4b88c1140.slice/crio-26b73c6e30955162450cd10199a9324913ccca0eb4131b6882d01c8210631b00 WatchSource:0}: Error finding container 26b73c6e30955162450cd10199a9324913ccca0eb4131b6882d01c8210631b00: Status 404 returned error can't find the container with id 26b73c6e30955162450cd10199a9324913ccca0eb4131b6882d01c8210631b00
Sep 17 19:37:41.382903 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:41.382863    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"]
Sep 17 19:37:41.386811 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:37:41.386783    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1400cba_fe90_4ff1_93fa_f90c315f721f.slice/crio-4be79105b142909c7c42faf21ef7863b7568177feb95c0e9034751e5115352c1 WatchSource:0}: Error finding container 4be79105b142909c7c42faf21ef7863b7568177feb95c0e9034751e5115352c1: Status 404 returned error can't find the container with id 4be79105b142909c7c42faf21ef7863b7568177feb95c0e9034751e5115352c1
Sep 17 19:37:41.486014 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:41.485976    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"1b874cca-d362-4260-980e-24e4b88c1140","Type":"ContainerStarted","Data":"c87220377d0ed44658684308ab24abe9e21dd3119256696607148f1a1930e88d"}
Sep 17 19:37:41.486213 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:41.486051    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"1b874cca-d362-4260-980e-24e4b88c1140","Type":"ContainerStarted","Data":"26b73c6e30955162450cd10199a9324913ccca0eb4131b6882d01c8210631b00"}
Sep 17 19:37:41.491291 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:41.491261    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8" event={"ID":"f1400cba-fe90-4ff1-93fa-f90c315f721f","Type":"ContainerStarted","Data":"8970546c113128674d117855fff4271531189b75c8ca3ee7cc799488e77ed972"}
Sep 17 19:37:41.491421 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:41.491312    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8" event={"ID":"f1400cba-fe90-4ff1-93fa-f90c315f721f","Type":"ContainerStarted","Data":"4be79105b142909c7c42faf21ef7863b7568177feb95c0e9034751e5115352c1"}
Sep 17 19:37:41.491421 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:41.491333    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"
Sep 17 19:37:44.510113 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:44.510080    2587 generic.go:356] "Generic (PLEG): container finished" podID="1b874cca-d362-4260-980e-24e4b88c1140" containerID="c87220377d0ed44658684308ab24abe9e21dd3119256696607148f1a1930e88d" exitCode=0
Sep 17 19:37:44.510523 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:44.510157    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"1b874cca-d362-4260-980e-24e4b88c1140","Type":"ContainerDied","Data":"c87220377d0ed44658684308ab24abe9e21dd3119256696607148f1a1930e88d"}
Sep 17 19:37:44.535178 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:44.535127    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8" podStartSLOduration=4.535110241 podStartE2EDuration="4.535110241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:37:41.585689753 +0000 UTC m=+3513.464658507" watchObservedRunningTime="2026-09-17 19:37:44.535110241 +0000 UTC m=+3516.414078998"
Sep 17 19:37:44.701345 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:44.701304    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"]
Sep 17 19:37:44.706780 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:44.706747    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-nsgfv"]
Sep 17 19:37:45.257430 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:45.257387    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:45.330351 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:45.330312    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:45.430382 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:45.430346    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:45.497146 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:45.497113    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:45.517645 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:45.517560    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"1b874cca-d362-4260-980e-24e4b88c1140","Type":"ContainerStarted","Data":"a5f3d07b9248e606ce862174a6c206d9f429649a6a05cc32ab1b9643d1017e14"}
Sep 17 19:37:45.543478 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:45.543422    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=3.265129702 podStartE2EDuration="5.543406556s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.278276854s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:37:45.540574832 +0000 UTC m=+3517.419543602" watchObservedRunningTime="2026-09-17 19:37:45.543406556 +0000 UTC m=+3517.422375313"
Sep 17 19:37:45.852994 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:45.852889    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:46.258455 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:46.258422    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99937b63-2642-4100-beec-008ca8776d30" path="/var/lib/kubelet/pods/99937b63-2642-4100-beec-008ca8776d30/volumes"
Sep 17 19:37:51.524332 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:51.524294    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:51.598010 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:37:51.597977    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:38:07.329768 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:07.329728    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:38:10.659729 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:10.659689    2587 scope.go:117] "RemoveContainer" containerID="133e986d0cdfb7f5d36b87b06ef780e88fb81392f0f543d487737e4c54a4c7ac"
Sep 17 19:38:12.500421 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:12.500388    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"
Sep 17 19:38:17.681523 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:17.681488    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:38:18.202013 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:18.201980    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:38:29.455335 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.455287    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:38:29.646693 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.646643    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"]
Sep 17 19:38:29.650261 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.650234    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"
Sep 17 19:38:29.654144 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.654119    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 19:38:29.654291 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.654133    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-ltq47\""
Sep 17 19:38:29.654291 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.654171    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-tfbsr\""
Sep 17 19:38:29.654291 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.654186    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 19:38:29.654291 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.654134    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 19:38:29.662160 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.662130    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"]
Sep 17 19:38:29.685631 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.685588    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:38:29.737118 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.737087    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zd7d\" (UniqueName: \"kubernetes.io/projected/f0f82442-1896-4e8e-a5de-ab1dc2603c37-kube-api-access-9zd7d\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4\" (UID: \"f0f82442-1896-4e8e-a5de-ab1dc2603c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"
Sep 17 19:38:29.737286 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.737165    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0f82442-1896-4e8e-a5de-ab1dc2603c37-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4\" (UID: \"f0f82442-1896-4e8e-a5de-ab1dc2603c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"
Sep 17 19:38:29.838504 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.838466    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zd7d\" (UniqueName: \"kubernetes.io/projected/f0f82442-1896-4e8e-a5de-ab1dc2603c37-kube-api-access-9zd7d\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4\" (UID: \"f0f82442-1896-4e8e-a5de-ab1dc2603c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"
Sep 17 19:38:29.838704 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.838553    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0f82442-1896-4e8e-a5de-ab1dc2603c37-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4\" (UID: \"f0f82442-1896-4e8e-a5de-ab1dc2603c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"
Sep 17 19:38:29.841512 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.841473    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0f82442-1896-4e8e-a5de-ab1dc2603c37-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4\" (UID: \"f0f82442-1896-4e8e-a5de-ab1dc2603c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"
Sep 17 19:38:29.848260 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.848217    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zd7d\" (UniqueName: \"kubernetes.io/projected/f0f82442-1896-4e8e-a5de-ab1dc2603c37-kube-api-access-9zd7d\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4\" (UID: \"f0f82442-1896-4e8e-a5de-ab1dc2603c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"
Sep 17 19:38:29.963352 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:29.963314    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"
Sep 17 19:38:30.239841 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:30.239662    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"]
Sep 17 19:38:30.243415 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:38:30.243384    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f82442_1896_4e8e_a5de_ab1dc2603c37.slice/crio-de2de060e8824b3f0ad6e2c60abcf77b02690a22870908d646bcd16a2a1d2809 WatchSource:0}: Error finding container de2de060e8824b3f0ad6e2c60abcf77b02690a22870908d646bcd16a2a1d2809: Status 404 returned error can't find the container with id de2de060e8824b3f0ad6e2c60abcf77b02690a22870908d646bcd16a2a1d2809
Sep 17 19:38:30.746461 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:30.746424    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4" event={"ID":"f0f82442-1896-4e8e-a5de-ab1dc2603c37","Type":"ContainerStarted","Data":"c78ebc629d138c95a94402d4e9fa0adcd35d6dce03e81e074f8f1f7487955d86"}
Sep 17 19:38:30.746874 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:30.746514    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"
Sep 17 19:38:30.746874 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:30.746535    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4" event={"ID":"f0f82442-1896-4e8e-a5de-ab1dc2603c37","Type":"ContainerStarted","Data":"de2de060e8824b3f0ad6e2c60abcf77b02690a22870908d646bcd16a2a1d2809"}
Sep 17 19:38:30.784083 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:30.784027    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4" podStartSLOduration=1.7840120339999999 podStartE2EDuration="1.784012034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:38:30.783451702 +0000 UTC m=+3562.662420459" watchObservedRunningTime="2026-09-17 19:38:30.784012034 +0000 UTC m=+3562.662980791"
Sep 17 19:38:46.904975 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:38:46.904912    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:39:01.753901 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:39:01.753869    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"
Sep 17 19:39:10.470973 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:39:10.470923    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:39:10.472880 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:39:10.472858    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:39:37.259617 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:39:37.259573    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:39:37.277763 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:39:37.277721    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"]
Sep 17 19:39:37.336360 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:39:37.336311    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:39:42.499476 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:39:42.499436    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8" podUID="f1400cba-fe90-4ff1-93fa-f90c315f721f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:39:52.298081 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:39:52.298037    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8" podUID="f1400cba-fe90-4ff1-93fa-f90c315f721f" containerName="main" containerID="cri-o://8970546c113128674d117855fff4271531189b75c8ca3ee7cc799488e77ed972" gracePeriod=45
Sep 17 19:39:52.499566 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:39:52.499524    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8" podUID="f1400cba-fe90-4ff1-93fa-f90c315f721f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:39:54.027712 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:39:54.027676    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 19:40:02.499782 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:02.499740    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8" podUID="f1400cba-fe90-4ff1-93fa-f90c315f721f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:40:02.500240 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:02.499845    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"
Sep 17 19:40:09.050245 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.050174    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="1b874cca-d362-4260-980e-24e4b88c1140" containerName="main" containerID="cri-o://a5f3d07b9248e606ce862174a6c206d9f429649a6a05cc32ab1b9643d1017e14" gracePeriod=45
Sep 17 19:40:09.733932 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.733896    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:40:09.789776 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.789731    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-kserve-provision-location\") pod \"1b874cca-d362-4260-980e-24e4b88c1140\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") "
Sep 17 19:40:09.790007 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.789857    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-model-cache\") pod \"1b874cca-d362-4260-980e-24e4b88c1140\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") "
Sep 17 19:40:09.790007 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.789923    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b874cca-d362-4260-980e-24e4b88c1140-tls-certs\") pod \"1b874cca-d362-4260-980e-24e4b88c1140\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") "
Sep 17 19:40:09.790007 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.789967    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b874cca-d362-4260-980e-24e4b88c1140-cabundle-cert\") pod \"1b874cca-d362-4260-980e-24e4b88c1140\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") "
Sep 17 19:40:09.790007 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.789990    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-home\") pod \"1b874cca-d362-4260-980e-24e4b88c1140\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") "
Sep 17 19:40:09.790239 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.790010    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-tmp-dir\") pod \"1b874cca-d362-4260-980e-24e4b88c1140\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") "
Sep 17 19:40:09.790239 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.790034    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmnvh\" (UniqueName: \"kubernetes.io/projected/1b874cca-d362-4260-980e-24e4b88c1140-kube-api-access-dmnvh\") pod \"1b874cca-d362-4260-980e-24e4b88c1140\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") "
Sep 17 19:40:09.790239 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.790069    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-dshm\") pod \"1b874cca-d362-4260-980e-24e4b88c1140\" (UID: \"1b874cca-d362-4260-980e-24e4b88c1140\") "
Sep 17 19:40:09.790239 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.790190    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b874cca-d362-4260-980e-24e4b88c1140" (UID: "1b874cca-d362-4260-980e-24e4b88c1140"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:40:09.790452 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.790323    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:40:09.790452 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.790403    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b874cca-d362-4260-980e-24e4b88c1140-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1b874cca-d362-4260-980e-24e4b88c1140" (UID: "1b874cca-d362-4260-980e-24e4b88c1140"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:40:09.790649 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.790622    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-home" (OuterVolumeSpecName: "home") pod "1b874cca-d362-4260-980e-24e4b88c1140" (UID: "1b874cca-d362-4260-980e-24e4b88c1140"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:40:09.792707 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.792681    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-dshm" (OuterVolumeSpecName: "dshm") pod "1b874cca-d362-4260-980e-24e4b88c1140" (UID: "1b874cca-d362-4260-980e-24e4b88c1140"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:40:09.793060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.793039    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b874cca-d362-4260-980e-24e4b88c1140-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b874cca-d362-4260-980e-24e4b88c1140" (UID: "1b874cca-d362-4260-980e-24e4b88c1140"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:40:09.793115 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.793079    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b874cca-d362-4260-980e-24e4b88c1140-kube-api-access-dmnvh" (OuterVolumeSpecName: "kube-api-access-dmnvh") pod "1b874cca-d362-4260-980e-24e4b88c1140" (UID: "1b874cca-d362-4260-980e-24e4b88c1140"). InnerVolumeSpecName "kube-api-access-dmnvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:40:09.803103 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.803068    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b874cca-d362-4260-980e-24e4b88c1140" (UID: "1b874cca-d362-4260-980e-24e4b88c1140"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:40:09.815882 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.815829    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b874cca-d362-4260-980e-24e4b88c1140" (UID: "1b874cca-d362-4260-980e-24e4b88c1140"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:40:09.891536 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.891498    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b874cca-d362-4260-980e-24e4b88c1140-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:40:09.891536 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.891528    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b874cca-d362-4260-980e-24e4b88c1140-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:40:09.891536 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.891536    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:40:09.891536 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.891543    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:40:09.891809 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.891551    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmnvh\" (UniqueName: \"kubernetes.io/projected/1b874cca-d362-4260-980e-24e4b88c1140-kube-api-access-dmnvh\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:40:09.891809 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.891560    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:40:09.891809 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:09.891569    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b874cca-d362-4260-980e-24e4b88c1140-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:40:10.196160 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:10.196065    2587 generic.go:356] "Generic (PLEG): container finished" podID="1b874cca-d362-4260-980e-24e4b88c1140" containerID="a5f3d07b9248e606ce862174a6c206d9f429649a6a05cc32ab1b9643d1017e14" exitCode=0
Sep 17 19:40:10.196160 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:10.196115    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:40:10.196634 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:10.196156    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"1b874cca-d362-4260-980e-24e4b88c1140","Type":"ContainerDied","Data":"a5f3d07b9248e606ce862174a6c206d9f429649a6a05cc32ab1b9643d1017e14"}
Sep 17 19:40:10.196634 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:10.196213    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"1b874cca-d362-4260-980e-24e4b88c1140","Type":"ContainerDied","Data":"26b73c6e30955162450cd10199a9324913ccca0eb4131b6882d01c8210631b00"}
Sep 17 19:40:10.196634 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:10.196235    2587 scope.go:117] "RemoveContainer" containerID="a5f3d07b9248e606ce862174a6c206d9f429649a6a05cc32ab1b9643d1017e14"
Sep 17 19:40:10.206170 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:10.206148    2587 scope.go:117] "RemoveContainer" containerID="c87220377d0ed44658684308ab24abe9e21dd3119256696607148f1a1930e88d"
Sep 17 19:40:10.216173 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:10.216154    2587 scope.go:117] "RemoveContainer" containerID="a5f3d07b9248e606ce862174a6c206d9f429649a6a05cc32ab1b9643d1017e14"
Sep 17 19:40:10.216471 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:40:10.216452    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f3d07b9248e606ce862174a6c206d9f429649a6a05cc32ab1b9643d1017e14\": container with ID starting with a5f3d07b9248e606ce862174a6c206d9f429649a6a05cc32ab1b9643d1017e14 not found: ID does not exist" containerID="a5f3d07b9248e606ce862174a6c206d9f429649a6a05cc32ab1b9643d1017e14"
Sep 17 19:40:10.216510 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:10.216480    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f3d07b9248e606ce862174a6c206d9f429649a6a05cc32ab1b9643d1017e14"} err="failed to get container status \"a5f3d07b9248e606ce862174a6c206d9f429649a6a05cc32ab1b9643d1017e14\": rpc error: code = NotFound desc = could not find container \"a5f3d07b9248e606ce862174a6c206d9f429649a6a05cc32ab1b9643d1017e14\": container with ID starting with a5f3d07b9248e606ce862174a6c206d9f429649a6a05cc32ab1b9643d1017e14 not found: ID does not exist"
Sep 17 19:40:10.216510 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:10.216498    2587 scope.go:117] "RemoveContainer" containerID="c87220377d0ed44658684308ab24abe9e21dd3119256696607148f1a1930e88d"
Sep 17 19:40:10.216747 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:40:10.216725    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87220377d0ed44658684308ab24abe9e21dd3119256696607148f1a1930e88d\": container with ID starting with c87220377d0ed44658684308ab24abe9e21dd3119256696607148f1a1930e88d not found: ID does not exist" containerID="c87220377d0ed44658684308ab24abe9e21dd3119256696607148f1a1930e88d"
Sep 17 19:40:10.216784 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:10.216755    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87220377d0ed44658684308ab24abe9e21dd3119256696607148f1a1930e88d"} err="failed to get container status \"c87220377d0ed44658684308ab24abe9e21dd3119256696607148f1a1930e88d\": rpc error: code = NotFound desc = could not find container \"c87220377d0ed44658684308ab24abe9e21dd3119256696607148f1a1930e88d\": container with ID starting with c87220377d0ed44658684308ab24abe9e21dd3119256696607148f1a1930e88d not found: ID does not exist"
Sep 17 19:40:10.219405 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:10.219382    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 19:40:10.223266 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:10.223241    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 19:40:10.258444 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:10.258411    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b874cca-d362-4260-980e-24e4b88c1140" path="/var/lib/kubelet/pods/1b874cca-d362-4260-980e-24e4b88c1140/volumes"
Sep 17 19:40:12.500457 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:12.500416    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8" podUID="f1400cba-fe90-4ff1-93fa-f90c315f721f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:40:22.558395 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:22.558322    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"
Sep 17 19:40:22.700456 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:22.700421    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnf6g\" (UniqueName: \"kubernetes.io/projected/f1400cba-fe90-4ff1-93fa-f90c315f721f-kube-api-access-hnf6g\") pod \"f1400cba-fe90-4ff1-93fa-f90c315f721f\" (UID: \"f1400cba-fe90-4ff1-93fa-f90c315f721f\") "
Sep 17 19:40:22.700672 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:22.700501    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1400cba-fe90-4ff1-93fa-f90c315f721f-tls-certs\") pod \"f1400cba-fe90-4ff1-93fa-f90c315f721f\" (UID: \"f1400cba-fe90-4ff1-93fa-f90c315f721f\") "
Sep 17 19:40:22.702833 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:22.702802    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1400cba-fe90-4ff1-93fa-f90c315f721f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1400cba-fe90-4ff1-93fa-f90c315f721f" (UID: "f1400cba-fe90-4ff1-93fa-f90c315f721f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:40:22.702968 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:22.702830    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1400cba-fe90-4ff1-93fa-f90c315f721f-kube-api-access-hnf6g" (OuterVolumeSpecName: "kube-api-access-hnf6g") pod "f1400cba-fe90-4ff1-93fa-f90c315f721f" (UID: "f1400cba-fe90-4ff1-93fa-f90c315f721f"). InnerVolumeSpecName "kube-api-access-hnf6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:40:22.801523 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:22.801477    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnf6g\" (UniqueName: \"kubernetes.io/projected/f1400cba-fe90-4ff1-93fa-f90c315f721f-kube-api-access-hnf6g\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:40:22.801523 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:22.801513    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1400cba-fe90-4ff1-93fa-f90c315f721f-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:40:23.261629 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:23.261590    2587 generic.go:356] "Generic (PLEG): container finished" podID="f1400cba-fe90-4ff1-93fa-f90c315f721f" containerID="8970546c113128674d117855fff4271531189b75c8ca3ee7cc799488e77ed972" exitCode=0
Sep 17 19:40:23.261629 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:23.261627    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"
Sep 17 19:40:23.261849 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:23.261669    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8" event={"ID":"f1400cba-fe90-4ff1-93fa-f90c315f721f","Type":"ContainerDied","Data":"8970546c113128674d117855fff4271531189b75c8ca3ee7cc799488e77ed972"}
Sep 17 19:40:23.261849 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:23.261714    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8" event={"ID":"f1400cba-fe90-4ff1-93fa-f90c315f721f","Type":"ContainerDied","Data":"4be79105b142909c7c42faf21ef7863b7568177feb95c0e9034751e5115352c1"}
Sep 17 19:40:23.261849 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:23.261735    2587 scope.go:117] "RemoveContainer" containerID="8970546c113128674d117855fff4271531189b75c8ca3ee7cc799488e77ed972"
Sep 17 19:40:23.270991 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:23.270968    2587 scope.go:117] "RemoveContainer" containerID="8970546c113128674d117855fff4271531189b75c8ca3ee7cc799488e77ed972"
Sep 17 19:40:23.271221 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:40:23.271204    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8970546c113128674d117855fff4271531189b75c8ca3ee7cc799488e77ed972\": container with ID starting with 8970546c113128674d117855fff4271531189b75c8ca3ee7cc799488e77ed972 not found: ID does not exist" containerID="8970546c113128674d117855fff4271531189b75c8ca3ee7cc799488e77ed972"
Sep 17 19:40:23.271272 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:23.271228    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8970546c113128674d117855fff4271531189b75c8ca3ee7cc799488e77ed972"} err="failed to get container status \"8970546c113128674d117855fff4271531189b75c8ca3ee7cc799488e77ed972\": rpc error: code = NotFound desc = could not find container \"8970546c113128674d117855fff4271531189b75c8ca3ee7cc799488e77ed972\": container with ID starting with 8970546c113128674d117855fff4271531189b75c8ca3ee7cc799488e77ed972 not found: ID does not exist"
Sep 17 19:40:23.291441 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:23.291410    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"]
Sep 17 19:40:23.296694 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:23.296666    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8"]
Sep 17 19:40:23.499191 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:23.499143    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemj4v8" podUID="f1400cba-fe90-4ff1-93fa-f90c315f721f" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.58:9003\" within 1s: context deadline exceeded"
Sep 17 19:40:23.499429 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:40:23.499236    2587 logging.go:55] [core] [Channel #353 SubChannel #354]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.58:9003", ServerName: "10.132.0.58:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.58:9003: operation was canceled"
Sep 17 19:40:24.257530 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:24.257488    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1400cba-fe90-4ff1-93fa-f90c315f721f" path="/var/lib/kubelet/pods/f1400cba-fe90-4ff1-93fa-f90c315f721f/volumes"
Sep 17 19:40:27.175197 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:27.175161    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:40:27.271856 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:27.271816    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:40:33.577486 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:33.577451    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:40:39.212538 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:39.212500    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:40:39.282550 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:39.282515    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:40:44.418816 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:44.418781    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:40:44.658623 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:44.658593    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:40:55.514322 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:40:55.514288    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:02.349169 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:02.349129    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:02.349607 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:02.349374    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"]
Sep 17 19:41:02.429706 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:02.429670    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:10.701471 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:10.701434    2587 scope.go:117] "RemoveContainer" containerID="fc1b351ff5dd83fb4eb5d012b9dfc5d62a8afa7a10334e56d429eaab10b69ba4"
Sep 17 19:41:11.754318 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:11.754271    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4" podUID="f0f82442-1896-4e8e-a5de-ab1dc2603c37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:41:17.369621 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:17.369574    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4" podUID="f0f82442-1896-4e8e-a5de-ab1dc2603c37" containerName="main" containerID="cri-o://c78ebc629d138c95a94402d4e9fa0adcd35d6dce03e81e074f8f1f7487955d86" gracePeriod=45
Sep 17 19:41:18.767838 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:18.767799    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:18.862509 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:18.862454    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:21.753455 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:21.753414    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4" podUID="f0f82442-1896-4e8e-a5de-ab1dc2603c37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:41:31.754145 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:31.754102    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4" podUID="f0f82442-1896-4e8e-a5de-ab1dc2603c37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:41:31.754642 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:31.754222    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"
Sep 17 19:41:41.753100 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:41.753056    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4" podUID="f0f82442-1896-4e8e-a5de-ab1dc2603c37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:41:47.624777 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.624744    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"
Sep 17 19:41:47.634074 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.633993    2587 generic.go:356] "Generic (PLEG): container finished" podID="f0f82442-1896-4e8e-a5de-ab1dc2603c37" containerID="c78ebc629d138c95a94402d4e9fa0adcd35d6dce03e81e074f8f1f7487955d86" exitCode=0
Sep 17 19:41:47.634074 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.634038    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"
Sep 17 19:41:47.634251 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.634072    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4" event={"ID":"f0f82442-1896-4e8e-a5de-ab1dc2603c37","Type":"ContainerDied","Data":"c78ebc629d138c95a94402d4e9fa0adcd35d6dce03e81e074f8f1f7487955d86"}
Sep 17 19:41:47.634251 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.634152    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4" event={"ID":"f0f82442-1896-4e8e-a5de-ab1dc2603c37","Type":"ContainerDied","Data":"de2de060e8824b3f0ad6e2c60abcf77b02690a22870908d646bcd16a2a1d2809"}
Sep 17 19:41:47.634251 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.634165    2587 scope.go:117] "RemoveContainer" containerID="c78ebc629d138c95a94402d4e9fa0adcd35d6dce03e81e074f8f1f7487955d86"
Sep 17 19:41:47.643335 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.643311    2587 scope.go:117] "RemoveContainer" containerID="c78ebc629d138c95a94402d4e9fa0adcd35d6dce03e81e074f8f1f7487955d86"
Sep 17 19:41:47.643663 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:41:47.643640    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c78ebc629d138c95a94402d4e9fa0adcd35d6dce03e81e074f8f1f7487955d86\": container with ID starting with c78ebc629d138c95a94402d4e9fa0adcd35d6dce03e81e074f8f1f7487955d86 not found: ID does not exist" containerID="c78ebc629d138c95a94402d4e9fa0adcd35d6dce03e81e074f8f1f7487955d86"
Sep 17 19:41:47.643732 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.643675    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78ebc629d138c95a94402d4e9fa0adcd35d6dce03e81e074f8f1f7487955d86"} err="failed to get container status \"c78ebc629d138c95a94402d4e9fa0adcd35d6dce03e81e074f8f1f7487955d86\": rpc error: code = NotFound desc = could not find container \"c78ebc629d138c95a94402d4e9fa0adcd35d6dce03e81e074f8f1f7487955d86\": container with ID starting with c78ebc629d138c95a94402d4e9fa0adcd35d6dce03e81e074f8f1f7487955d86 not found: ID does not exist"
Sep 17 19:41:47.704325 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.704283    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zd7d\" (UniqueName: \"kubernetes.io/projected/f0f82442-1896-4e8e-a5de-ab1dc2603c37-kube-api-access-9zd7d\") pod \"f0f82442-1896-4e8e-a5de-ab1dc2603c37\" (UID: \"f0f82442-1896-4e8e-a5de-ab1dc2603c37\") "
Sep 17 19:41:47.704513 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.704343    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0f82442-1896-4e8e-a5de-ab1dc2603c37-tls-certs\") pod \"f0f82442-1896-4e8e-a5de-ab1dc2603c37\" (UID: \"f0f82442-1896-4e8e-a5de-ab1dc2603c37\") "
Sep 17 19:41:47.706822 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.706793    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0f82442-1896-4e8e-a5de-ab1dc2603c37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0f82442-1896-4e8e-a5de-ab1dc2603c37" (UID: "f0f82442-1896-4e8e-a5de-ab1dc2603c37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:41:47.706822 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.706806    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f82442-1896-4e8e-a5de-ab1dc2603c37-kube-api-access-9zd7d" (OuterVolumeSpecName: "kube-api-access-9zd7d") pod "f0f82442-1896-4e8e-a5de-ab1dc2603c37" (UID: "f0f82442-1896-4e8e-a5de-ab1dc2603c37"). InnerVolumeSpecName "kube-api-access-9zd7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:41:47.805450 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.805395    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zd7d\" (UniqueName: \"kubernetes.io/projected/f0f82442-1896-4e8e-a5de-ab1dc2603c37-kube-api-access-9zd7d\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:41:47.805450 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.805444    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0f82442-1896-4e8e-a5de-ab1dc2603c37-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:41:47.956213 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.956176    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"]
Sep 17 19:41:47.961571 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:47.961545    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg6p4"]
Sep 17 19:41:48.257923 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:48.257887    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0f82442-1896-4e8e-a5de-ab1dc2603c37" path="/var/lib/kubelet/pods/f0f82442-1896-4e8e-a5de-ab1dc2603c37/volumes"
Sep 17 19:41:52.461163 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:52.461071    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:52.563913 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:52.563876    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:57.425696 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:57.425661    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:58.741405 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:41:58.741366    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:04.575062 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:04.575023    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:04.649890 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:04.649862    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:08.578219 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:08.578181    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:09.765824 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:09.765785    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:09.855626 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:09.855587    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:10.175024 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.174917    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"]
Sep 17 19:42:10.175299 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.175284    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1400cba-fe90-4ff1-93fa-f90c315f721f" containerName="main"
Sep 17 19:42:10.175343 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.175301    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1400cba-fe90-4ff1-93fa-f90c315f721f" containerName="main"
Sep 17 19:42:10.175343 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.175309    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f82442-1896-4e8e-a5de-ab1dc2603c37" containerName="main"
Sep 17 19:42:10.175343 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.175315    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f82442-1896-4e8e-a5de-ab1dc2603c37" containerName="main"
Sep 17 19:42:10.175343 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.175327    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b874cca-d362-4260-980e-24e4b88c1140" containerName="storage-initializer"
Sep 17 19:42:10.175343 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.175334    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b874cca-d362-4260-980e-24e4b88c1140" containerName="storage-initializer"
Sep 17 19:42:10.175343 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.175339    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b874cca-d362-4260-980e-24e4b88c1140" containerName="main"
Sep 17 19:42:10.175343 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.175344    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b874cca-d362-4260-980e-24e4b88c1140" containerName="main"
Sep 17 19:42:10.175556 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.175409    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1400cba-fe90-4ff1-93fa-f90c315f721f" containerName="main"
Sep 17 19:42:10.175556 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.175416    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b874cca-d362-4260-980e-24e4b88c1140" containerName="main"
Sep 17 19:42:10.175556 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.175426    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0f82442-1896-4e8e-a5de-ab1dc2603c37" containerName="main"
Sep 17 19:42:10.179734 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.179710    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"
Sep 17 19:42:10.183803 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.183772    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-fzw5l\""
Sep 17 19:42:10.183989 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.183877    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 19:42:10.183989 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.183789    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 19:42:10.183989 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.183816    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 19:42:10.184183 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.183789    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-2rb6s\""
Sep 17 19:42:10.191715 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.191687    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"]
Sep 17 19:42:10.214793 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.214760    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:10.285763 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.285709    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c66gf\" (UniqueName: \"kubernetes.io/projected/5c4611ca-f398-4815-8b0e-a0fb7e73e40f-kube-api-access-c66gf\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7\" (UID: \"5c4611ca-f398-4815-8b0e-a0fb7e73e40f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"
Sep 17 19:42:10.285987 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.285794    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4611ca-f398-4815-8b0e-a0fb7e73e40f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7\" (UID: \"5c4611ca-f398-4815-8b0e-a0fb7e73e40f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"
Sep 17 19:42:10.386340 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.386307    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c66gf\" (UniqueName: \"kubernetes.io/projected/5c4611ca-f398-4815-8b0e-a0fb7e73e40f-kube-api-access-c66gf\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7\" (UID: \"5c4611ca-f398-4815-8b0e-a0fb7e73e40f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"
Sep 17 19:42:10.386552 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.386497    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4611ca-f398-4815-8b0e-a0fb7e73e40f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7\" (UID: \"5c4611ca-f398-4815-8b0e-a0fb7e73e40f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"
Sep 17 19:42:10.389379 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.389350    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4611ca-f398-4815-8b0e-a0fb7e73e40f-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7\" (UID: \"5c4611ca-f398-4815-8b0e-a0fb7e73e40f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"
Sep 17 19:42:10.394842 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.394795    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c66gf\" (UniqueName: \"kubernetes.io/projected/5c4611ca-f398-4815-8b0e-a0fb7e73e40f-kube-api-access-c66gf\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7\" (UID: \"5c4611ca-f398-4815-8b0e-a0fb7e73e40f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"
Sep 17 19:42:10.492104 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.492065    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"
Sep 17 19:42:10.772692 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:10.772664    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"]
Sep 17 19:42:10.774242 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:42:10.774214    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4611ca_f398_4815_8b0e_a0fb7e73e40f.slice/crio-0b32eb598c4ef1a3fc64229cfa572e592471fefdf118ed2c81737223eecfbf15 WatchSource:0}: Error finding container 0b32eb598c4ef1a3fc64229cfa572e592471fefdf118ed2c81737223eecfbf15: Status 404 returned error can't find the container with id 0b32eb598c4ef1a3fc64229cfa572e592471fefdf118ed2c81737223eecfbf15
Sep 17 19:42:11.740904 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:11.740862    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7" event={"ID":"5c4611ca-f398-4815-8b0e-a0fb7e73e40f","Type":"ContainerStarted","Data":"bdca90276f3c1a1141040b5c6cff224f53d268b1ea42a5824bd5d082c970efb8"}
Sep 17 19:42:11.741079 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:11.740926    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7" event={"ID":"5c4611ca-f398-4815-8b0e-a0fb7e73e40f","Type":"ContainerStarted","Data":"0b32eb598c4ef1a3fc64229cfa572e592471fefdf118ed2c81737223eecfbf15"}
Sep 17 19:42:11.741079 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:11.740972    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"
Sep 17 19:42:11.761870 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:11.761815    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7" podStartSLOduration=1.7617997600000002 podStartE2EDuration="1.76179976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:42:11.760644405 +0000 UTC m=+3783.639613162" watchObservedRunningTime="2026-09-17 19:42:11.76179976 +0000 UTC m=+3783.640768516"
Sep 17 19:42:15.063004 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:15.062966    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:20.704809 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:20.704773    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:20.773101 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:20.773059    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:21.446714 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:21.446683    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:25.984126 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:25.984085    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"]
Sep 17 19:42:25.987373 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:25.987348    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:25.989813 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:25.989785    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 19:42:25.990825 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:25.990802    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-lmv8l\""
Sep 17 19:42:25.990989 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:25.990808    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 19:42:25.990989 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:25.990839    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 19:42:25.998636 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:25.998610    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"]
Sep 17 19:42:26.101691 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.101654    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:26.112918 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.112882    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm8qk\" (UniqueName: \"kubernetes.io/projected/06bab65d-1c32-4c07-b4bb-276cce966e0c-kube-api-access-jm8qk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.113060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.112931    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.113060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.113011    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.113060 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.113040    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.113176 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.113100    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.113176 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.113117    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bab65d-1c32-4c07-b4bb-276cce966e0c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.213853 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.213808    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.213853 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.213853    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bab65d-1c32-4c07-b4bb-276cce966e0c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.214156 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.213898    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm8qk\" (UniqueName: \"kubernetes.io/projected/06bab65d-1c32-4c07-b4bb-276cce966e0c-kube-api-access-jm8qk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.214156 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.213962    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.214156 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.214000    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.214156 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.214036    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.214427 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.214406    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.214485 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.214452    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.214485 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.214463    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.216508 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.216488    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.216821 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.216800    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bab65d-1c32-4c07-b4bb-276cce966e0c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.225104 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.225074    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm8qk\" (UniqueName: \"kubernetes.io/projected/06bab65d-1c32-4c07-b4bb-276cce966e0c-kube-api-access-jm8qk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.287534 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.287446    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:26.299396 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.299356    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:26.572917 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.572883    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"]
Sep 17 19:42:26.575617 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:42:26.575589    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bab65d_1c32_4c07_b4bb_276cce966e0c.slice/crio-38974818aab902c2da4b85c85098507b7655d57e9f24f10bc9077cc65210f536 WatchSource:0}: Error finding container 38974818aab902c2da4b85c85098507b7655d57e9f24f10bc9077cc65210f536: Status 404 returned error can't find the container with id 38974818aab902c2da4b85c85098507b7655d57e9f24f10bc9077cc65210f536
Sep 17 19:42:26.803963 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.803893    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47" event={"ID":"06bab65d-1c32-4c07-b4bb-276cce966e0c","Type":"ContainerStarted","Data":"23d574ec86ec0208886176212571d60a2a32c41feb39a0b5d03efc90995e943a"}
Sep 17 19:42:26.804153 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.803981    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47" event={"ID":"06bab65d-1c32-4c07-b4bb-276cce966e0c","Type":"ContainerStarted","Data":"38974818aab902c2da4b85c85098507b7655d57e9f24f10bc9077cc65210f536"}
Sep 17 19:42:26.822671 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:26.822572    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47" podStartSLOduration=1.8225533170000001 podStartE2EDuration="1.822553317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:42:26.822302665 +0000 UTC m=+3798.701271435" watchObservedRunningTime="2026-09-17 19:42:26.822553317 +0000 UTC m=+3798.701522073"
Sep 17 19:42:27.299862 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:27.299824    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:34.246979 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:34.246920    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:34.311418 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:34.311382    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:36.300221 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:36.300180    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:36.312719 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:36.312693    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:36.860149 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:36.860115    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:42:39.078127 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:39.078089    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:42.747342 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:42:42.747309    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"
Sep 17 19:43:00.469713 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:00.469670    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"]
Sep 17 19:43:00.479381 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:43:00.479340    2587 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:43:00.479381 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:43:00.479381 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:43:00.479381 ip-10-0-140-194 kubenswrapper[2587]:  > containerID="23d574ec86ec0208886176212571d60a2a32c41feb39a0b5d03efc90995e943a" cmd=["/bin/sleep","15"]
Sep 17 19:43:00.479675 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:00.479398    2587 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47" message="" err=<
Sep 17 19:43:00.479675 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:43:00.479675 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:43:00.479675 ip-10-0-140-194 kubenswrapper[2587]:  >
Sep 17 19:43:00.479675 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:43:00.479410    2587 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:43:00.479675 ip-10-0-140-194 kubenswrapper[2587]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:43:00.479675 ip-10-0-140-194 kubenswrapper[2587]:         , stdout: , stderr: , exit code -1
Sep 17 19:43:00.479675 ip-10-0-140-194 kubenswrapper[2587]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47" podUID="06bab65d-1c32-4c07-b4bb-276cce966e0c" containerName="main" containerID="cri-o://23d574ec86ec0208886176212571d60a2a32c41feb39a0b5d03efc90995e943a"
Sep 17 19:43:00.479675 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:00.479449    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47" podUID="06bab65d-1c32-4c07-b4bb-276cce966e0c" containerName="main" containerID="cri-o://23d574ec86ec0208886176212571d60a2a32c41feb39a0b5d03efc90995e943a" gracePeriod=60
Sep 17 19:43:00.499037 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:00.498998    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:43:00.592526 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:00.592495    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:43:00.849856 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:00.849813    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47" podUID="06bab65d-1c32-4c07-b4bb-276cce966e0c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 17 19:43:00.954579 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:00.954544    2587 generic.go:356] "Generic (PLEG): container finished" podID="06bab65d-1c32-4c07-b4bb-276cce966e0c" containerID="23d574ec86ec0208886176212571d60a2a32c41feb39a0b5d03efc90995e943a" exitCode=0
Sep 17 19:43:00.954747 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:00.954601    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47" event={"ID":"06bab65d-1c32-4c07-b4bb-276cce966e0c","Type":"ContainerDied","Data":"23d574ec86ec0208886176212571d60a2a32c41feb39a0b5d03efc90995e943a"}
Sep 17 19:43:01.231991 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.231962    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:43:01.295394 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.295362    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-dshm\") pod \"06bab65d-1c32-4c07-b4bb-276cce966e0c\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") "
Sep 17 19:43:01.295589 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.295405    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jm8qk\" (UniqueName: \"kubernetes.io/projected/06bab65d-1c32-4c07-b4bb-276cce966e0c-kube-api-access-jm8qk\") pod \"06bab65d-1c32-4c07-b4bb-276cce966e0c\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") "
Sep 17 19:43:01.295589 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.295437    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bab65d-1c32-4c07-b4bb-276cce966e0c-tls-certs\") pod \"06bab65d-1c32-4c07-b4bb-276cce966e0c\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") "
Sep 17 19:43:01.295589 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.295456    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-model-cache\") pod \"06bab65d-1c32-4c07-b4bb-276cce966e0c\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") "
Sep 17 19:43:01.295589 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.295508    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-tmp-dir\") pod \"06bab65d-1c32-4c07-b4bb-276cce966e0c\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") "
Sep 17 19:43:01.295589 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.295534    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-home\") pod \"06bab65d-1c32-4c07-b4bb-276cce966e0c\" (UID: \"06bab65d-1c32-4c07-b4bb-276cce966e0c\") "
Sep 17 19:43:01.295857 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.295776    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "06bab65d-1c32-4c07-b4bb-276cce966e0c" (UID: "06bab65d-1c32-4c07-b4bb-276cce966e0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:43:01.295857 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.295792    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06bab65d-1c32-4c07-b4bb-276cce966e0c" (UID: "06bab65d-1c32-4c07-b4bb-276cce966e0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:43:01.295931 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.295862    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-home" (OuterVolumeSpecName: "home") pod "06bab65d-1c32-4c07-b4bb-276cce966e0c" (UID: "06bab65d-1c32-4c07-b4bb-276cce966e0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:43:01.297922 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.297888    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06bab65d-1c32-4c07-b4bb-276cce966e0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06bab65d-1c32-4c07-b4bb-276cce966e0c" (UID: "06bab65d-1c32-4c07-b4bb-276cce966e0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:43:01.298065 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.297957    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-dshm" (OuterVolumeSpecName: "dshm") pod "06bab65d-1c32-4c07-b4bb-276cce966e0c" (UID: "06bab65d-1c32-4c07-b4bb-276cce966e0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:43:01.298065 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.297981    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bab65d-1c32-4c07-b4bb-276cce966e0c-kube-api-access-jm8qk" (OuterVolumeSpecName: "kube-api-access-jm8qk") pod "06bab65d-1c32-4c07-b4bb-276cce966e0c" (UID: "06bab65d-1c32-4c07-b4bb-276cce966e0c"). InnerVolumeSpecName "kube-api-access-jm8qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:43:01.396986 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.396853    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bab65d-1c32-4c07-b4bb-276cce966e0c-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:43:01.396986 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.396897    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:43:01.396986 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.396926    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:43:01.396986 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.396933    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:43:01.396986 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.396966    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bab65d-1c32-4c07-b4bb-276cce966e0c-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:43:01.396986 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.396975    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm8qk\" (UniqueName: \"kubernetes.io/projected/06bab65d-1c32-4c07-b4bb-276cce966e0c-kube-api-access-jm8qk\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:43:01.960564 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.960529    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"
Sep 17 19:43:01.960564 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.960547    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47" event={"ID":"06bab65d-1c32-4c07-b4bb-276cce966e0c","Type":"ContainerDied","Data":"38974818aab902c2da4b85c85098507b7655d57e9f24f10bc9077cc65210f536"}
Sep 17 19:43:01.961169 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.960604    2587 scope.go:117] "RemoveContainer" containerID="23d574ec86ec0208886176212571d60a2a32c41feb39a0b5d03efc90995e943a"
Sep 17 19:43:01.982760 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.982724    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"]
Sep 17 19:43:01.989021 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:01.988991    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fnlb47"]
Sep 17 19:43:02.258183 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:02.258145    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06bab65d-1c32-4c07-b4bb-276cce966e0c" path="/var/lib/kubelet/pods/06bab65d-1c32-4c07-b4bb-276cce966e0c/volumes"
Sep 17 19:43:50.515322 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:50.515285    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:43:50.619633 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:50.619593    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:43:56.608318 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:43:56.608287    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:02.104423 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:02.104365    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:02.178509 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:02.178478    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:10.499162 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:10.499132    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:44:10.503501 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:10.503475    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:44:16.112137 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:16.112100    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:16.303491 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:16.303456    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:23.461379 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.461346    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"]
Sep 17 19:44:23.461888 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.461676    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06bab65d-1c32-4c07-b4bb-276cce966e0c" containerName="main"
Sep 17 19:44:23.461888 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.461687    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="06bab65d-1c32-4c07-b4bb-276cce966e0c" containerName="main"
Sep 17 19:44:23.461888 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.461750    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="06bab65d-1c32-4c07-b4bb-276cce966e0c" containerName="main"
Sep 17 19:44:23.464793 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.464772    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.467725 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.467698    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:44:23.467859 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.467721    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 19:44:23.478934 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.478904    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"]
Sep 17 19:44:23.551216 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.551171    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae3deb4d-d86d-4312-885e-72517be1394a-cabundle-cert\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.551216 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.551213    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-home\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.551440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.551274    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-model-cache\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.551440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.551302    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-tmp-dir\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.551440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.551328    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mvqx\" (UniqueName: \"kubernetes.io/projected/ae3deb4d-d86d-4312-885e-72517be1394a-kube-api-access-9mvqx\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.551440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.551364    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3deb4d-d86d-4312-885e-72517be1394a-tls-certs\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.551574 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.551436    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-kserve-provision-location\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.551574 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.551471    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-dshm\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.651851 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.651805    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-model-cache\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.652081 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.651860    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-tmp-dir\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.652081 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.651885    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mvqx\" (UniqueName: \"kubernetes.io/projected/ae3deb4d-d86d-4312-885e-72517be1394a-kube-api-access-9mvqx\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.652081 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.651963    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3deb4d-d86d-4312-885e-72517be1394a-tls-certs\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.652081 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.651999    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-kserve-provision-location\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.652081 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.652022    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-dshm\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.652081 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.652057    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae3deb4d-d86d-4312-885e-72517be1394a-cabundle-cert\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.652081 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.652080    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-home\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.652441 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.652419    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-model-cache\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.652509 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.652454    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-tmp-dir\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.652509 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.652487    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-kserve-provision-location\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.652609 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.652542    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-home\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.652815 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.652795    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae3deb4d-d86d-4312-885e-72517be1394a-cabundle-cert\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.654527 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.654503    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-dshm\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.654795 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.654777    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3deb4d-d86d-4312-885e-72517be1394a-tls-certs\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.661043 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.661020    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mvqx\" (UniqueName: \"kubernetes.io/projected/ae3deb4d-d86d-4312-885e-72517be1394a-kube-api-access-9mvqx\") pod \"prestop-hook-test-kserve-b9ddcc467-mpvrg\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:23.775079 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:23.775022    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:24.053771 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:24.053699    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"]
Sep 17 19:44:24.056643 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:44:24.056611    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3deb4d_d86d_4312_885e_72517be1394a.slice/crio-c35ff3dedaed7fac53acc06edbd11a63e6352c7c6072771c1c828494d952cb27 WatchSource:0}: Error finding container c35ff3dedaed7fac53acc06edbd11a63e6352c7c6072771c1c828494d952cb27: Status 404 returned error can't find the container with id c35ff3dedaed7fac53acc06edbd11a63e6352c7c6072771c1c828494d952cb27
Sep 17 19:44:24.322999 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:24.322892    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg" event={"ID":"ae3deb4d-d86d-4312-885e-72517be1394a","Type":"ContainerStarted","Data":"c5c4eb58cab2d078acf07087c39a4228ed3bf0e34317950708623174463b17f4"}
Sep 17 19:44:24.322999 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:24.322988    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg" event={"ID":"ae3deb4d-d86d-4312-885e-72517be1394a","Type":"ContainerStarted","Data":"c35ff3dedaed7fac53acc06edbd11a63e6352c7c6072771c1c828494d952cb27"}
Sep 17 19:44:26.333417 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:26.333384    2587 generic.go:356] "Generic (PLEG): container finished" podID="ae3deb4d-d86d-4312-885e-72517be1394a" containerID="c5c4eb58cab2d078acf07087c39a4228ed3bf0e34317950708623174463b17f4" exitCode=0
Sep 17 19:44:26.333786 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:26.333445    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg" event={"ID":"ae3deb4d-d86d-4312-885e-72517be1394a","Type":"ContainerDied","Data":"c5c4eb58cab2d078acf07087c39a4228ed3bf0e34317950708623174463b17f4"}
Sep 17 19:44:27.339652 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:27.339601    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg" event={"ID":"ae3deb4d-d86d-4312-885e-72517be1394a","Type":"ContainerStarted","Data":"57c98412472da571d800ee65ddaf6054c7854f16bdfefdfcb25969724916ef95"}
Sep 17 19:44:27.362435 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:27.362376    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg" podStartSLOduration=2.167957193 podStartE2EDuration="4.362358545s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.194401352s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:44:27.361429937 +0000 UTC m=+3919.240398736" watchObservedRunningTime="2026-09-17 19:44:27.362358545 +0000 UTC m=+3919.241327302"
Sep 17 19:44:27.775509 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:27.775473    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:32.581036 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:32.581001    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:33.775328 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:33.775290    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:44:33.776922 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:33.776894    2587 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg" podUID="ae3deb4d-d86d-4312-885e-72517be1394a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 17 19:44:41.825177 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:41.825140    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"]
Sep 17 19:44:41.830281 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:41.830252    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:41.838859 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:41.838827    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"]
Sep 17 19:44:41.899346 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:41.899306    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:42.747607 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:42.747562    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7" podUID="5c4611ca-f398-4815-8b0e-a0fb7e73e40f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:44:50.087685 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:50.087595    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:50.181624 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:50.181586    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:52.747705 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:52.747662    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7" podUID="5c4611ca-f398-4815-8b0e-a0fb7e73e40f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:44:56.846994 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:56.846916    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7" podUID="5c4611ca-f398-4815-8b0e-a0fb7e73e40f" containerName="main" containerID="cri-o://bdca90276f3c1a1141040b5c6cff224f53d268b1ea42a5824bd5d082c970efb8" gracePeriod=45
Sep 17 19:44:56.861568 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:44:56.861504    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg" podUID="ae3deb4d-d86d-4312-885e-72517be1394a" containerName="main" containerID="cri-o://57c98412472da571d800ee65ddaf6054c7854f16bdfefdfcb25969724916ef95" gracePeriod=45
Sep 17 19:45:02.747247 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:02.747207    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7" podUID="5c4611ca-f398-4815-8b0e-a0fb7e73e40f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:45:02.747688 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:02.747326    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"
Sep 17 19:45:12.747352 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:12.747312    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7" podUID="5c4611ca-f398-4815-8b0e-a0fb7e73e40f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:45:20.155611 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:20.155575    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:45:22.747550 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:22.747509    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7" podUID="5c4611ca-f398-4815-8b0e-a0fb7e73e40f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:45:27.099166 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.099139    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"
Sep 17 19:45:27.193513 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.193408    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c66gf\" (UniqueName: \"kubernetes.io/projected/5c4611ca-f398-4815-8b0e-a0fb7e73e40f-kube-api-access-c66gf\") pod \"5c4611ca-f398-4815-8b0e-a0fb7e73e40f\" (UID: \"5c4611ca-f398-4815-8b0e-a0fb7e73e40f\") "
Sep 17 19:45:27.193513 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.193463    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4611ca-f398-4815-8b0e-a0fb7e73e40f-tls-certs\") pod \"5c4611ca-f398-4815-8b0e-a0fb7e73e40f\" (UID: \"5c4611ca-f398-4815-8b0e-a0fb7e73e40f\") "
Sep 17 19:45:27.196054 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.196008    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4611ca-f398-4815-8b0e-a0fb7e73e40f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c4611ca-f398-4815-8b0e-a0fb7e73e40f" (UID: "5c4611ca-f398-4815-8b0e-a0fb7e73e40f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:45:27.196054 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.196020    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4611ca-f398-4815-8b0e-a0fb7e73e40f-kube-api-access-c66gf" (OuterVolumeSpecName: "kube-api-access-c66gf") pod "5c4611ca-f398-4815-8b0e-a0fb7e73e40f" (UID: "5c4611ca-f398-4815-8b0e-a0fb7e73e40f"). InnerVolumeSpecName "kube-api-access-c66gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:45:27.295096 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.295058    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c66gf\" (UniqueName: \"kubernetes.io/projected/5c4611ca-f398-4815-8b0e-a0fb7e73e40f-kube-api-access-c66gf\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:45:27.295096 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.295092    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c4611ca-f398-4815-8b0e-a0fb7e73e40f-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:45:27.623537 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.623503    2587 generic.go:356] "Generic (PLEG): container finished" podID="5c4611ca-f398-4815-8b0e-a0fb7e73e40f" containerID="bdca90276f3c1a1141040b5c6cff224f53d268b1ea42a5824bd5d082c970efb8" exitCode=0
Sep 17 19:45:27.623799 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.623542    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"
Sep 17 19:45:27.623799 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.623546    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7" event={"ID":"5c4611ca-f398-4815-8b0e-a0fb7e73e40f","Type":"ContainerDied","Data":"bdca90276f3c1a1141040b5c6cff224f53d268b1ea42a5824bd5d082c970efb8"}
Sep 17 19:45:27.623799 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.623683    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7" event={"ID":"5c4611ca-f398-4815-8b0e-a0fb7e73e40f","Type":"ContainerDied","Data":"0b32eb598c4ef1a3fc64229cfa572e592471fefdf118ed2c81737223eecfbf15"}
Sep 17 19:45:27.623799 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.623711    2587 scope.go:117] "RemoveContainer" containerID="bdca90276f3c1a1141040b5c6cff224f53d268b1ea42a5824bd5d082c970efb8"
Sep 17 19:45:27.633477 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.633450    2587 scope.go:117] "RemoveContainer" containerID="bdca90276f3c1a1141040b5c6cff224f53d268b1ea42a5824bd5d082c970efb8"
Sep 17 19:45:27.633795 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:45:27.633775    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdca90276f3c1a1141040b5c6cff224f53d268b1ea42a5824bd5d082c970efb8\": container with ID starting with bdca90276f3c1a1141040b5c6cff224f53d268b1ea42a5824bd5d082c970efb8 not found: ID does not exist" containerID="bdca90276f3c1a1141040b5c6cff224f53d268b1ea42a5824bd5d082c970efb8"
Sep 17 19:45:27.633867 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.633807    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdca90276f3c1a1141040b5c6cff224f53d268b1ea42a5824bd5d082c970efb8"} err="failed to get container status \"bdca90276f3c1a1141040b5c6cff224f53d268b1ea42a5824bd5d082c970efb8\": rpc error: code = NotFound desc = could not find container \"bdca90276f3c1a1141040b5c6cff224f53d268b1ea42a5824bd5d082c970efb8\": container with ID starting with bdca90276f3c1a1141040b5c6cff224f53d268b1ea42a5824bd5d082c970efb8 not found: ID does not exist"
Sep 17 19:45:27.644628 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.644600    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"]
Sep 17 19:45:27.651090 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:27.651065    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tbvs7"]
Sep 17 19:45:28.259609 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:28.259568    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c4611ca-f398-4815-8b0e-a0fb7e73e40f" path="/var/lib/kubelet/pods/5c4611ca-f398-4815-8b0e-a0fb7e73e40f/volumes"
Sep 17 19:45:40.189783 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:40.189740    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:45:42.513331 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.513301    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-b9ddcc467-mpvrg_ae3deb4d-d86d-4312-885e-72517be1394a/main/0.log"
Sep 17 19:45:42.516049 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.516027    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:45:42.630668 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.630568    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-home\") pod \"ae3deb4d-d86d-4312-885e-72517be1394a\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") "
Sep 17 19:45:42.630668 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.630615    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3deb4d-d86d-4312-885e-72517be1394a-tls-certs\") pod \"ae3deb4d-d86d-4312-885e-72517be1394a\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") "
Sep 17 19:45:42.630668 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.630636    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mvqx\" (UniqueName: \"kubernetes.io/projected/ae3deb4d-d86d-4312-885e-72517be1394a-kube-api-access-9mvqx\") pod \"ae3deb4d-d86d-4312-885e-72517be1394a\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") "
Sep 17 19:45:42.630668 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.630664    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-tmp-dir\") pod \"ae3deb4d-d86d-4312-885e-72517be1394a\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") "
Sep 17 19:45:42.631049 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.630682    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-model-cache\") pod \"ae3deb4d-d86d-4312-885e-72517be1394a\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") "
Sep 17 19:45:42.631049 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.630717    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-dshm\") pod \"ae3deb4d-d86d-4312-885e-72517be1394a\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") "
Sep 17 19:45:42.631049 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.630739    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae3deb4d-d86d-4312-885e-72517be1394a-cabundle-cert\") pod \"ae3deb4d-d86d-4312-885e-72517be1394a\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") "
Sep 17 19:45:42.631049 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.630754    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-kserve-provision-location\") pod \"ae3deb4d-d86d-4312-885e-72517be1394a\" (UID: \"ae3deb4d-d86d-4312-885e-72517be1394a\") "
Sep 17 19:45:42.631286 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.631206    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-home" (OuterVolumeSpecName: "home") pod "ae3deb4d-d86d-4312-885e-72517be1394a" (UID: "ae3deb4d-d86d-4312-885e-72517be1394a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:45:42.631286 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.631242    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae3deb4d-d86d-4312-885e-72517be1394a" (UID: "ae3deb4d-d86d-4312-885e-72517be1394a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:45:42.631286 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.631271    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae3deb4d-d86d-4312-885e-72517be1394a" (UID: "ae3deb4d-d86d-4312-885e-72517be1394a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:45:42.631493 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.631296    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3deb4d-d86d-4312-885e-72517be1394a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae3deb4d-d86d-4312-885e-72517be1394a" (UID: "ae3deb4d-d86d-4312-885e-72517be1394a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:45:42.633210 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.633183    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-dshm" (OuterVolumeSpecName: "dshm") pod "ae3deb4d-d86d-4312-885e-72517be1394a" (UID: "ae3deb4d-d86d-4312-885e-72517be1394a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:45:42.633315 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.633209    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3deb4d-d86d-4312-885e-72517be1394a-kube-api-access-9mvqx" (OuterVolumeSpecName: "kube-api-access-9mvqx") pod "ae3deb4d-d86d-4312-885e-72517be1394a" (UID: "ae3deb4d-d86d-4312-885e-72517be1394a"). InnerVolumeSpecName "kube-api-access-9mvqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:45:42.633377 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.633360    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae3deb4d-d86d-4312-885e-72517be1394a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae3deb4d-d86d-4312-885e-72517be1394a" (UID: "ae3deb4d-d86d-4312-885e-72517be1394a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:45:42.655741 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.655683    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae3deb4d-d86d-4312-885e-72517be1394a" (UID: "ae3deb4d-d86d-4312-885e-72517be1394a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:45:42.703406 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.703374    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-b9ddcc467-mpvrg_ae3deb4d-d86d-4312-885e-72517be1394a/main/0.log"
Sep 17 19:45:42.703795 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.703771    2587 generic.go:356] "Generic (PLEG): container finished" podID="ae3deb4d-d86d-4312-885e-72517be1394a" containerID="57c98412472da571d800ee65ddaf6054c7854f16bdfefdfcb25969724916ef95" exitCode=137
Sep 17 19:45:42.703885 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.703826    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"
Sep 17 19:45:42.703885 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.703861    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg" event={"ID":"ae3deb4d-d86d-4312-885e-72517be1394a","Type":"ContainerDied","Data":"57c98412472da571d800ee65ddaf6054c7854f16bdfefdfcb25969724916ef95"}
Sep 17 19:45:42.703994 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.703917    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg" event={"ID":"ae3deb4d-d86d-4312-885e-72517be1394a","Type":"ContainerDied","Data":"c35ff3dedaed7fac53acc06edbd11a63e6352c7c6072771c1c828494d952cb27"}
Sep 17 19:45:42.703994 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.703964    2587 scope.go:117] "RemoveContainer" containerID="57c98412472da571d800ee65ddaf6054c7854f16bdfefdfcb25969724916ef95"
Sep 17 19:45:42.714293 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.714271    2587 scope.go:117] "RemoveContainer" containerID="c5c4eb58cab2d078acf07087c39a4228ed3bf0e34317950708623174463b17f4"
Sep 17 19:45:42.722779 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.722756    2587 scope.go:117] "RemoveContainer" containerID="57c98412472da571d800ee65ddaf6054c7854f16bdfefdfcb25969724916ef95"
Sep 17 19:45:42.723097 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:45:42.723073    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c98412472da571d800ee65ddaf6054c7854f16bdfefdfcb25969724916ef95\": container with ID starting with 57c98412472da571d800ee65ddaf6054c7854f16bdfefdfcb25969724916ef95 not found: ID does not exist" containerID="57c98412472da571d800ee65ddaf6054c7854f16bdfefdfcb25969724916ef95"
Sep 17 19:45:42.723167 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.723105    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c98412472da571d800ee65ddaf6054c7854f16bdfefdfcb25969724916ef95"} err="failed to get container status \"57c98412472da571d800ee65ddaf6054c7854f16bdfefdfcb25969724916ef95\": rpc error: code = NotFound desc = could not find container \"57c98412472da571d800ee65ddaf6054c7854f16bdfefdfcb25969724916ef95\": container with ID starting with 57c98412472da571d800ee65ddaf6054c7854f16bdfefdfcb25969724916ef95 not found: ID does not exist"
Sep 17 19:45:42.723167 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.723125    2587 scope.go:117] "RemoveContainer" containerID="c5c4eb58cab2d078acf07087c39a4228ed3bf0e34317950708623174463b17f4"
Sep 17 19:45:42.723398 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:45:42.723380    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c4eb58cab2d078acf07087c39a4228ed3bf0e34317950708623174463b17f4\": container with ID starting with c5c4eb58cab2d078acf07087c39a4228ed3bf0e34317950708623174463b17f4 not found: ID does not exist" containerID="c5c4eb58cab2d078acf07087c39a4228ed3bf0e34317950708623174463b17f4"
Sep 17 19:45:42.723440 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.723406    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c4eb58cab2d078acf07087c39a4228ed3bf0e34317950708623174463b17f4"} err="failed to get container status \"c5c4eb58cab2d078acf07087c39a4228ed3bf0e34317950708623174463b17f4\": rpc error: code = NotFound desc = could not find container \"c5c4eb58cab2d078acf07087c39a4228ed3bf0e34317950708623174463b17f4\": container with ID starting with c5c4eb58cab2d078acf07087c39a4228ed3bf0e34317950708623174463b17f4 not found: ID does not exist"
Sep 17 19:45:42.728145 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.728122    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"]
Sep 17 19:45:42.731636 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.731611    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3deb4d-d86d-4312-885e-72517be1394a-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:45:42.731727 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.731641    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mvqx\" (UniqueName: \"kubernetes.io/projected/ae3deb4d-d86d-4312-885e-72517be1394a-kube-api-access-9mvqx\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:45:42.731727 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.731656    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-tmp-dir\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:45:42.731727 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.731669    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-model-cache\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:45:42.731727 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.731681    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-dshm\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:45:42.731727 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.731693    2587 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae3deb4d-d86d-4312-885e-72517be1394a-cabundle-cert\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:45:42.731727 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.731706    2587 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-kserve-provision-location\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:45:42.731727 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.731717    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae3deb4d-d86d-4312-885e-72517be1394a-home\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:45:42.732036 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:42.732018    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b9ddcc467-mpvrg"]
Sep 17 19:45:44.257755 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:44.257717    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae3deb4d-d86d-4312-885e-72517be1394a" path="/var/lib/kubelet/pods/ae3deb4d-d86d-4312-885e-72517be1394a/volumes"
Sep 17 19:45:46.291924 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:46.291893    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:45:46.980870 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:46.980834    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:45:47.190852 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:47.190811    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:45:51.875768 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:51.875735    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:45:51.975494 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:51.975465    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:45:53.195832 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:53.195758    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:45:58.777182 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:58.777148    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:45:58.877802 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:45:58.877766    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:46:18.291232 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.291198    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:46:18.463515 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.463482    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"]
Sep 17 19:46:18.463788 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.463775    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4611ca-f398-4815-8b0e-a0fb7e73e40f" containerName="main"
Sep 17 19:46:18.463837 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.463789    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4611ca-f398-4815-8b0e-a0fb7e73e40f" containerName="main"
Sep 17 19:46:18.463837 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.463804    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3deb4d-d86d-4312-885e-72517be1394a" containerName="storage-initializer"
Sep 17 19:46:18.463837 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.463810    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3deb4d-d86d-4312-885e-72517be1394a" containerName="storage-initializer"
Sep 17 19:46:18.463837 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.463819    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3deb4d-d86d-4312-885e-72517be1394a" containerName="main"
Sep 17 19:46:18.463837 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.463824    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3deb4d-d86d-4312-885e-72517be1394a" containerName="main"
Sep 17 19:46:18.464015 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.463889    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae3deb4d-d86d-4312-885e-72517be1394a" containerName="main"
Sep 17 19:46:18.464015 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.463897    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c4611ca-f398-4815-8b0e-a0fb7e73e40f" containerName="main"
Sep 17 19:46:18.466584 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.466565    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"
Sep 17 19:46:18.472231 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.472210    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 19:46:18.472231 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.472220    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 19:46:18.472417 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.472273    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-5srmv\""
Sep 17 19:46:18.472417 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.472223    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 19:46:18.472417 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.472374    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-dx5j7\""
Sep 17 19:46:18.489648 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.489619    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"]
Sep 17 19:46:18.511664 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.511632    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:46:18.531332 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.531299    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgfdj\" (UniqueName: \"kubernetes.io/projected/cff59dbd-e19c-4736-97c0-8f283dcf3905-kube-api-access-fgfdj\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr\" (UID: \"cff59dbd-e19c-4736-97c0-8f283dcf3905\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"
Sep 17 19:46:18.531468 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.531350    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff59dbd-e19c-4736-97c0-8f283dcf3905-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr\" (UID: \"cff59dbd-e19c-4736-97c0-8f283dcf3905\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"
Sep 17 19:46:18.632109 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.632017    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgfdj\" (UniqueName: \"kubernetes.io/projected/cff59dbd-e19c-4736-97c0-8f283dcf3905-kube-api-access-fgfdj\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr\" (UID: \"cff59dbd-e19c-4736-97c0-8f283dcf3905\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"
Sep 17 19:46:18.632109 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.632066    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff59dbd-e19c-4736-97c0-8f283dcf3905-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr\" (UID: \"cff59dbd-e19c-4736-97c0-8f283dcf3905\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"
Sep 17 19:46:18.634707 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.634677    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff59dbd-e19c-4736-97c0-8f283dcf3905-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr\" (UID: \"cff59dbd-e19c-4736-97c0-8f283dcf3905\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"
Sep 17 19:46:18.644394 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.644373    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgfdj\" (UniqueName: \"kubernetes.io/projected/cff59dbd-e19c-4736-97c0-8f283dcf3905-kube-api-access-fgfdj\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr\" (UID: \"cff59dbd-e19c-4736-97c0-8f283dcf3905\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"
Sep 17 19:46:18.776746 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:18.776703    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"
Sep 17 19:46:19.055078 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:19.054980    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"]
Sep 17 19:46:19.057607 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:46:19.057577    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff59dbd_e19c_4736_97c0_8f283dcf3905.slice/crio-ef508326d2c0b9d23d4571684b15c3a7843382e739ca859f5cfe4670208f07e7 WatchSource:0}: Error finding container ef508326d2c0b9d23d4571684b15c3a7843382e739ca859f5cfe4670208f07e7: Status 404 returned error can't find the container with id ef508326d2c0b9d23d4571684b15c3a7843382e739ca859f5cfe4670208f07e7
Sep 17 19:46:19.863779 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:19.863689    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr" event={"ID":"cff59dbd-e19c-4736-97c0-8f283dcf3905","Type":"ContainerStarted","Data":"24f66475ed0844b5e6b2278e3adec1ebf730613d1d50a529e147ec846e9e75da"}
Sep 17 19:46:19.863779 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:19.863754    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr" event={"ID":"cff59dbd-e19c-4736-97c0-8f283dcf3905","Type":"ContainerStarted","Data":"ef508326d2c0b9d23d4571684b15c3a7843382e739ca859f5cfe4670208f07e7"}
Sep 17 19:46:19.864257 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:19.863828    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"
Sep 17 19:46:19.881416 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:19.881359    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr" podStartSLOduration=1.881341795 podStartE2EDuration="1.881341795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:46:19.880970184 +0000 UTC m=+4031.759938942" watchObservedRunningTime="2026-09-17 19:46:19.881341795 +0000 UTC m=+4031.760310554"
Sep 17 19:46:32.019385 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:32.019343    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:46:50.871622 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:50.871589    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"
Sep 17 19:46:55.327506 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:55.327453    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"]
Sep 17 19:46:55.339860 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:55.339827    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:46:55.408964 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:46:55.408899    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:00.871295 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:00.871252    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr" podUID="cff59dbd-e19c-4736-97c0-8f283dcf3905" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:47:10.348243 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:10.348193    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr" podUID="cff59dbd-e19c-4736-97c0-8f283dcf3905" containerName="main" containerID="cri-o://24f66475ed0844b5e6b2278e3adec1ebf730613d1d50a529e147ec846e9e75da" gracePeriod=45
Sep 17 19:47:10.870967 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:10.870901    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr" podUID="cff59dbd-e19c-4736-97c0-8f283dcf3905" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:47:20.871697 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:20.871653    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr" podUID="cff59dbd-e19c-4736-97c0-8f283dcf3905" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:47:20.872150 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:20.871755    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"
Sep 17 19:47:22.482346 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:22.482302    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:24.029890 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:24.029855    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:30.871696 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:30.871652    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr" podUID="cff59dbd-e19c-4736-97c0-8f283dcf3905" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:47:40.607761 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:40.607700    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"
Sep 17 19:47:40.627031 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:40.626999    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff59dbd-e19c-4736-97c0-8f283dcf3905-tls-certs\") pod \"cff59dbd-e19c-4736-97c0-8f283dcf3905\" (UID: \"cff59dbd-e19c-4736-97c0-8f283dcf3905\") "
Sep 17 19:47:40.627218 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:40.627108    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgfdj\" (UniqueName: \"kubernetes.io/projected/cff59dbd-e19c-4736-97c0-8f283dcf3905-kube-api-access-fgfdj\") pod \"cff59dbd-e19c-4736-97c0-8f283dcf3905\" (UID: \"cff59dbd-e19c-4736-97c0-8f283dcf3905\") "
Sep 17 19:47:40.629641 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:40.629605    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff59dbd-e19c-4736-97c0-8f283dcf3905-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cff59dbd-e19c-4736-97c0-8f283dcf3905" (UID: "cff59dbd-e19c-4736-97c0-8f283dcf3905"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:47:40.629847 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:40.629826    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff59dbd-e19c-4736-97c0-8f283dcf3905-kube-api-access-fgfdj" (OuterVolumeSpecName: "kube-api-access-fgfdj") pod "cff59dbd-e19c-4736-97c0-8f283dcf3905" (UID: "cff59dbd-e19c-4736-97c0-8f283dcf3905"). InnerVolumeSpecName "kube-api-access-fgfdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:47:40.728205 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:40.728165    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgfdj\" (UniqueName: \"kubernetes.io/projected/cff59dbd-e19c-4736-97c0-8f283dcf3905-kube-api-access-fgfdj\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:47:40.728205 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:40.728197    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff59dbd-e19c-4736-97c0-8f283dcf3905-tls-certs\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:47:41.229064 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:41.229023    2587 generic.go:356] "Generic (PLEG): container finished" podID="cff59dbd-e19c-4736-97c0-8f283dcf3905" containerID="24f66475ed0844b5e6b2278e3adec1ebf730613d1d50a529e147ec846e9e75da" exitCode=0
Sep 17 19:47:41.229267 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:41.229072    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"
Sep 17 19:47:41.229267 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:41.229106    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr" event={"ID":"cff59dbd-e19c-4736-97c0-8f283dcf3905","Type":"ContainerDied","Data":"24f66475ed0844b5e6b2278e3adec1ebf730613d1d50a529e147ec846e9e75da"}
Sep 17 19:47:41.229267 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:41.229163    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr" event={"ID":"cff59dbd-e19c-4736-97c0-8f283dcf3905","Type":"ContainerDied","Data":"ef508326d2c0b9d23d4571684b15c3a7843382e739ca859f5cfe4670208f07e7"}
Sep 17 19:47:41.229267 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:41.229185    2587 scope.go:117] "RemoveContainer" containerID="24f66475ed0844b5e6b2278e3adec1ebf730613d1d50a529e147ec846e9e75da"
Sep 17 19:47:41.238057 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:41.238034    2587 scope.go:117] "RemoveContainer" containerID="24f66475ed0844b5e6b2278e3adec1ebf730613d1d50a529e147ec846e9e75da"
Sep 17 19:47:41.238386 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:47:41.238360    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f66475ed0844b5e6b2278e3adec1ebf730613d1d50a529e147ec846e9e75da\": container with ID starting with 24f66475ed0844b5e6b2278e3adec1ebf730613d1d50a529e147ec846e9e75da not found: ID does not exist" containerID="24f66475ed0844b5e6b2278e3adec1ebf730613d1d50a529e147ec846e9e75da"
Sep 17 19:47:41.238449 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:41.238398    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f66475ed0844b5e6b2278e3adec1ebf730613d1d50a529e147ec846e9e75da"} err="failed to get container status \"24f66475ed0844b5e6b2278e3adec1ebf730613d1d50a529e147ec846e9e75da\": rpc error: code = NotFound desc = could not find container \"24f66475ed0844b5e6b2278e3adec1ebf730613d1d50a529e147ec846e9e75da\": container with ID starting with 24f66475ed0844b5e6b2278e3adec1ebf730613d1d50a529e147ec846e9e75da not found: ID does not exist"
Sep 17 19:47:41.250745 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:41.250708    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"]
Sep 17 19:47:41.254501 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:41.254476    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zq2fr"]
Sep 17 19:47:42.258294 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:42.258258    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cff59dbd-e19c-4736-97c0-8f283dcf3905" path="/var/lib/kubelet/pods/cff59dbd-e19c-4736-97c0-8f283dcf3905/volumes"
Sep 17 19:47:44.173063 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:44.173024    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:45.369004 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.368966    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jwbv6/must-gather-mfzkd"]
Sep 17 19:47:45.369420 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.369280    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff59dbd-e19c-4736-97c0-8f283dcf3905" containerName="main"
Sep 17 19:47:45.369420 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.369291    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff59dbd-e19c-4736-97c0-8f283dcf3905" containerName="main"
Sep 17 19:47:45.369420 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.369356    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff59dbd-e19c-4736-97c0-8f283dcf3905" containerName="main"
Sep 17 19:47:45.373662 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.373642    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jwbv6/must-gather-mfzkd"
Sep 17 19:47:45.376639 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.376620    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jwbv6\"/\"kube-root-ca.crt\""
Sep 17 19:47:45.377687 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.377664    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jwbv6\"/\"openshift-service-ca.crt\""
Sep 17 19:47:45.377774 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.377693    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jwbv6\"/\"default-dockercfg-5lps7\""
Sep 17 19:47:45.384665 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.384637    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jwbv6/must-gather-mfzkd"]
Sep 17 19:47:45.387158 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.387133    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:45.463241 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.463199    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/97a48bab-afa4-4162-baa0-f6f762543808-must-gather-output\") pod \"must-gather-mfzkd\" (UID: \"97a48bab-afa4-4162-baa0-f6f762543808\") " pod="openshift-must-gather-jwbv6/must-gather-mfzkd"
Sep 17 19:47:45.463241 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.463251    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtjxr\" (UniqueName: \"kubernetes.io/projected/97a48bab-afa4-4162-baa0-f6f762543808-kube-api-access-rtjxr\") pod \"must-gather-mfzkd\" (UID: \"97a48bab-afa4-4162-baa0-f6f762543808\") " pod="openshift-must-gather-jwbv6/must-gather-mfzkd"
Sep 17 19:47:45.482397 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.482367    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:45.563692 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.563648    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/97a48bab-afa4-4162-baa0-f6f762543808-must-gather-output\") pod \"must-gather-mfzkd\" (UID: \"97a48bab-afa4-4162-baa0-f6f762543808\") " pod="openshift-must-gather-jwbv6/must-gather-mfzkd"
Sep 17 19:47:45.563878 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.563703    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtjxr\" (UniqueName: \"kubernetes.io/projected/97a48bab-afa4-4162-baa0-f6f762543808-kube-api-access-rtjxr\") pod \"must-gather-mfzkd\" (UID: \"97a48bab-afa4-4162-baa0-f6f762543808\") " pod="openshift-must-gather-jwbv6/must-gather-mfzkd"
Sep 17 19:47:45.564061 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.564040    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/97a48bab-afa4-4162-baa0-f6f762543808-must-gather-output\") pod \"must-gather-mfzkd\" (UID: \"97a48bab-afa4-4162-baa0-f6f762543808\") " pod="openshift-must-gather-jwbv6/must-gather-mfzkd"
Sep 17 19:47:45.575540 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.575502    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtjxr\" (UniqueName: \"kubernetes.io/projected/97a48bab-afa4-4162-baa0-f6f762543808-kube-api-access-rtjxr\") pod \"must-gather-mfzkd\" (UID: \"97a48bab-afa4-4162-baa0-f6f762543808\") " pod="openshift-must-gather-jwbv6/must-gather-mfzkd"
Sep 17 19:47:45.683049 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.682924    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jwbv6/must-gather-mfzkd"
Sep 17 19:47:45.955072 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.955044    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jwbv6/must-gather-mfzkd"]
Sep 17 19:47:45.957410 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:47:45.957383    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97a48bab_afa4_4162_baa0_f6f762543808.slice/crio-bf26377e10728d147dd28c25dc7155f5ff436c31bac5edf20fa19134a032aab8 WatchSource:0}: Error finding container bf26377e10728d147dd28c25dc7155f5ff436c31bac5edf20fa19134a032aab8: Status 404 returned error can't find the container with id bf26377e10728d147dd28c25dc7155f5ff436c31bac5edf20fa19134a032aab8
Sep 17 19:47:45.959163 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:45.959146    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:47:46.256921 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:46.256883    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jwbv6/must-gather-mfzkd" event={"ID":"97a48bab-afa4-4162-baa0-f6f762543808","Type":"ContainerStarted","Data":"bf26377e10728d147dd28c25dc7155f5ff436c31bac5edf20fa19134a032aab8"}
Sep 17 19:47:50.021703 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:50.021610    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:53.290338 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:53.290296    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jwbv6/must-gather-mfzkd" event={"ID":"97a48bab-afa4-4162-baa0-f6f762543808","Type":"ContainerStarted","Data":"980b4ac00f1737ac2f3798eec9ea08b644f8fbb46204e60c59c3fba1a98c9913"}
Sep 17 19:47:53.290774 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:53.290356    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jwbv6/must-gather-mfzkd" event={"ID":"97a48bab-afa4-4162-baa0-f6f762543808","Type":"ContainerStarted","Data":"48a0a706bd80c44b18902b5b75c1e5050cabc36c558eb22faec262ff51663800"}
Sep 17 19:47:53.307176 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:53.307104    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jwbv6/must-gather-mfzkd" podStartSLOduration=1.991203015 podStartE2EDuration="8.307077155s" totalImagesPullingTime="6.31587414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:47:53.306037474 +0000 UTC m=+4125.185006231" watchObservedRunningTime="2026-09-17 19:47:53.307077155 +0000 UTC m=+4125.186045913"
Sep 17 19:47:55.552995 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:55.552933    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:55.652391 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:47:55.652352    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:48:09.023493 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:09.023454    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7646b475dd-4rb55_ad67d410-cd73-46d3-81b8-d409d33ca41c/router/0.log"
Sep 17 19:48:09.841548 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:09.841517    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7646b475dd-4rb55_ad67d410-cd73-46d3-81b8-d409d33ca41c/router/0.log"
Sep 17 19:48:10.999671 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:10.999643    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-5tjqp_635a6392-7e09-4b14-8210-4cd7f8a88197/limitador/0.log"
Sep 17 19:48:15.401469 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:15.401434    2587 generic.go:356] "Generic (PLEG): container finished" podID="97a48bab-afa4-4162-baa0-f6f762543808" containerID="48a0a706bd80c44b18902b5b75c1e5050cabc36c558eb22faec262ff51663800" exitCode=0
Sep 17 19:48:15.401889 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:15.401504    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jwbv6/must-gather-mfzkd" event={"ID":"97a48bab-afa4-4162-baa0-f6f762543808","Type":"ContainerDied","Data":"48a0a706bd80c44b18902b5b75c1e5050cabc36c558eb22faec262ff51663800"}
Sep 17 19:48:15.401889 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:15.401826    2587 scope.go:117] "RemoveContainer" containerID="48a0a706bd80c44b18902b5b75c1e5050cabc36c558eb22faec262ff51663800"
Sep 17 19:48:16.191009 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:16.190975    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jwbv6_must-gather-mfzkd_97a48bab-afa4-4162-baa0-f6f762543808/gather/0.log"
Sep 17 19:48:16.630492 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:16.630461    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:48:16.780896 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:16.780868    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:48:16.884731 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:16.884661    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:48:19.729373 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:19.729340    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9sj77_ddd560fe-7729-4b3b-a307-d896e48b0baa/global-pull-secret-syncer/0.log"
Sep 17 19:48:19.834051 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:19.834025    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log"
Sep 17 19:48:20.012902 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:20.012868    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-194.ec2.internal_fdc7ceb412f9c95a22d608979ff76c2f/haproxy/0.log"
Sep 17 19:48:21.604521 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:21.604473    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-jwbv6/must-gather-mfzkd"]
Sep 17 19:48:21.604992 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:21.604668    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-jwbv6/must-gather-mfzkd" podUID="97a48bab-afa4-4162-baa0-f6f762543808" containerName="copy" containerID="cri-o://980b4ac00f1737ac2f3798eec9ea08b644f8fbb46204e60c59c3fba1a98c9913" gracePeriod=2
Sep 17 19:48:21.609219 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:21.608857    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-jwbv6/must-gather-mfzkd"]
Sep 17 19:48:21.841930 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:21.841906    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jwbv6_must-gather-mfzkd_97a48bab-afa4-4162-baa0-f6f762543808/copy/0.log"
Sep 17 19:48:21.843499 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:21.843480    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jwbv6/must-gather-mfzkd"
Sep 17 19:48:21.845561 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:48:21.845538    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-mfzkd\" is forbidden: User \"system:node:ip-10-0-140-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jwbv6\": no relationship found between node 'ip-10-0-140-194.ec2.internal' and this object" podUID="97a48bab-afa4-4162-baa0-f6f762543808" pod="openshift-must-gather-jwbv6/must-gather-mfzkd"
Sep 17 19:48:21.979102 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:21.979000    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/97a48bab-afa4-4162-baa0-f6f762543808-must-gather-output\") pod \"97a48bab-afa4-4162-baa0-f6f762543808\" (UID: \"97a48bab-afa4-4162-baa0-f6f762543808\") "
Sep 17 19:48:21.979102 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:21.979060    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtjxr\" (UniqueName: \"kubernetes.io/projected/97a48bab-afa4-4162-baa0-f6f762543808-kube-api-access-rtjxr\") pod \"97a48bab-afa4-4162-baa0-f6f762543808\" (UID: \"97a48bab-afa4-4162-baa0-f6f762543808\") "
Sep 17 19:48:21.981533 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:21.981497    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a48bab-afa4-4162-baa0-f6f762543808-kube-api-access-rtjxr" (OuterVolumeSpecName: "kube-api-access-rtjxr") pod "97a48bab-afa4-4162-baa0-f6f762543808" (UID: "97a48bab-afa4-4162-baa0-f6f762543808"). InnerVolumeSpecName "kube-api-access-rtjxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:48:21.986787 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:21.986754    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a48bab-afa4-4162-baa0-f6f762543808-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "97a48bab-afa4-4162-baa0-f6f762543808" (UID: "97a48bab-afa4-4162-baa0-f6f762543808"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:48:22.080417 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:22.080377    2587 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/97a48bab-afa4-4162-baa0-f6f762543808-must-gather-output\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:48:22.080417 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:22.080411    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtjxr\" (UniqueName: \"kubernetes.io/projected/97a48bab-afa4-4162-baa0-f6f762543808-kube-api-access-rtjxr\") on node \"ip-10-0-140-194.ec2.internal\" DevicePath \"\""
Sep 17 19:48:22.257773 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:22.257738    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97a48bab-afa4-4162-baa0-f6f762543808" path="/var/lib/kubelet/pods/97a48bab-afa4-4162-baa0-f6f762543808/volumes"
Sep 17 19:48:22.395827 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:22.395789    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:48:22.431027 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:22.430998    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jwbv6_must-gather-mfzkd_97a48bab-afa4-4162-baa0-f6f762543808/copy/0.log"
Sep 17 19:48:22.431413 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:22.431388    2587 generic.go:356] "Generic (PLEG): container finished" podID="97a48bab-afa4-4162-baa0-f6f762543808" containerID="980b4ac00f1737ac2f3798eec9ea08b644f8fbb46204e60c59c3fba1a98c9913" exitCode=143
Sep 17 19:48:22.431486 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:22.431451    2587 scope.go:117] "RemoveContainer" containerID="980b4ac00f1737ac2f3798eec9ea08b644f8fbb46204e60c59c3fba1a98c9913"
Sep 17 19:48:22.431486 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:22.431461    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jwbv6/must-gather-mfzkd"
Sep 17 19:48:22.439602 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:22.439580    2587 scope.go:117] "RemoveContainer" containerID="48a0a706bd80c44b18902b5b75c1e5050cabc36c558eb22faec262ff51663800"
Sep 17 19:48:22.453928 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:22.453905    2587 scope.go:117] "RemoveContainer" containerID="980b4ac00f1737ac2f3798eec9ea08b644f8fbb46204e60c59c3fba1a98c9913"
Sep 17 19:48:22.454328 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:48:22.454306    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980b4ac00f1737ac2f3798eec9ea08b644f8fbb46204e60c59c3fba1a98c9913\": container with ID starting with 980b4ac00f1737ac2f3798eec9ea08b644f8fbb46204e60c59c3fba1a98c9913 not found: ID does not exist" containerID="980b4ac00f1737ac2f3798eec9ea08b644f8fbb46204e60c59c3fba1a98c9913"
Sep 17 19:48:22.454375 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:22.454340    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980b4ac00f1737ac2f3798eec9ea08b644f8fbb46204e60c59c3fba1a98c9913"} err="failed to get container status \"980b4ac00f1737ac2f3798eec9ea08b644f8fbb46204e60c59c3fba1a98c9913\": rpc error: code = NotFound desc = could not find container \"980b4ac00f1737ac2f3798eec9ea08b644f8fbb46204e60c59c3fba1a98c9913\": container with ID starting with 980b4ac00f1737ac2f3798eec9ea08b644f8fbb46204e60c59c3fba1a98c9913 not found: ID does not exist"
Sep 17 19:48:22.454375 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:22.454360    2587 scope.go:117] "RemoveContainer" containerID="48a0a706bd80c44b18902b5b75c1e5050cabc36c558eb22faec262ff51663800"
Sep 17 19:48:22.454622 ip-10-0-140-194 kubenswrapper[2587]: E0917 19:48:22.454597    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a0a706bd80c44b18902b5b75c1e5050cabc36c558eb22faec262ff51663800\": container with ID starting with 48a0a706bd80c44b18902b5b75c1e5050cabc36c558eb22faec262ff51663800 not found: ID does not exist" containerID="48a0a706bd80c44b18902b5b75c1e5050cabc36c558eb22faec262ff51663800"
Sep 17 19:48:22.454682 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:22.454641    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a0a706bd80c44b18902b5b75c1e5050cabc36c558eb22faec262ff51663800"} err="failed to get container status \"48a0a706bd80c44b18902b5b75c1e5050cabc36c558eb22faec262ff51663800\": rpc error: code = NotFound desc = could not find container \"48a0a706bd80c44b18902b5b75c1e5050cabc36c558eb22faec262ff51663800\": container with ID starting with 48a0a706bd80c44b18902b5b75c1e5050cabc36c558eb22faec262ff51663800 not found: ID does not exist"
Sep 17 19:48:22.500668 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:22.500632    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:48:24.912437 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:24.912399    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-5tjqp_635a6392-7e09-4b14-8210-4cd7f8a88197/limitador/0.log"
Sep 17 19:48:26.466135 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:26.466104    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-qcr7l_1ca5000b-7618-4602-8388-1c272aa30ae8/cluster-monitoring-operator/0.log"
Sep 17 19:48:26.726860 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:26.726764    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:48:26.805010 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:26.804931    2587 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9nxm8_f28c7ec2-e7d8-441d-ae45-781ddd349850/konnectivity-agent/0.log" limit=1024
Sep 17 19:48:26.823208 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:26.823178    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wtwnk_3a8d6c0f-51b4-4eda-a99f-0708f00bd53d/node-exporter/0.log"
Sep 17 19:48:26.847001 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:26.846969    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wtwnk_3a8d6c0f-51b4-4eda-a99f-0708f00bd53d/kube-rbac-proxy/0.log"
Sep 17 19:48:26.873419 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:26.873393    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wtwnk_3a8d6c0f-51b4-4eda-a99f-0708f00bd53d/init-textfile/0.log"
Sep 17 19:48:28.985891 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:28.985855    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-872nz_01a4765a-fe1b-40b2-a611-5e815c42cc84/networking-console-plugin/0.log"
Sep 17 19:48:29.553260 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:29.553230    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/1.log"
Sep 17 19:48:29.558026 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:29.558005    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fm5ml_1e4b5a1a-b85d-4ff5-9741-99678d16aad1/console-operator/2.log"
Sep 17 19:48:30.054119 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.054083    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-4cd7g_c3d93259-1f5a-44c1-815f-ea577fbdabb7/download-server/0.log"
Sep 17 19:48:30.422373 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.422288    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"]
Sep 17 19:48:30.422617 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.422604    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a48bab-afa4-4162-baa0-f6f762543808" containerName="copy"
Sep 17 19:48:30.422660 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.422619    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a48bab-afa4-4162-baa0-f6f762543808" containerName="copy"
Sep 17 19:48:30.422660 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.422631    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a48bab-afa4-4162-baa0-f6f762543808" containerName="gather"
Sep 17 19:48:30.422660 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.422637    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a48bab-afa4-4162-baa0-f6f762543808" containerName="gather"
Sep 17 19:48:30.422758 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.422689    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a48bab-afa4-4162-baa0-f6f762543808" containerName="gather"
Sep 17 19:48:30.422758 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.422699    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a48bab-afa4-4162-baa0-f6f762543808" containerName="copy"
Sep 17 19:48:30.426363 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.426342    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.430066 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.430043    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7vd6\"/\"kube-root-ca.crt\""
Sep 17 19:48:30.430993 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.430972    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l7vd6\"/\"default-dockercfg-z24dx\""
Sep 17 19:48:30.431129 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.431000    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7vd6\"/\"openshift-service-ca.crt\""
Sep 17 19:48:30.442468 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.442430    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"]
Sep 17 19:48:30.547362 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.547322    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dx6w\" (UniqueName: \"kubernetes.io/projected/560d5b6a-2afe-4d73-aede-b1d458609827-kube-api-access-8dx6w\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.547564 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.547377    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/560d5b6a-2afe-4d73-aede-b1d458609827-lib-modules\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.547564 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.547432    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/560d5b6a-2afe-4d73-aede-b1d458609827-podres\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.547564 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.547504    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/560d5b6a-2afe-4d73-aede-b1d458609827-sys\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.547564 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.547532    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/560d5b6a-2afe-4d73-aede-b1d458609827-proc\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.648011 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.647968    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/560d5b6a-2afe-4d73-aede-b1d458609827-proc\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.648203 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.648021    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dx6w\" (UniqueName: \"kubernetes.io/projected/560d5b6a-2afe-4d73-aede-b1d458609827-kube-api-access-8dx6w\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.648203 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.648055    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/560d5b6a-2afe-4d73-aede-b1d458609827-lib-modules\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.648203 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.648075    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/560d5b6a-2afe-4d73-aede-b1d458609827-podres\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.648203 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.648085    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/560d5b6a-2afe-4d73-aede-b1d458609827-proc\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.648203 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.648140    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/560d5b6a-2afe-4d73-aede-b1d458609827-sys\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.648203 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.648196    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/560d5b6a-2afe-4d73-aede-b1d458609827-podres\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.648430 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.648209    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/560d5b6a-2afe-4d73-aede-b1d458609827-lib-modules\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.648430 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.648246    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/560d5b6a-2afe-4d73-aede-b1d458609827-sys\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.657500 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.657473    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dx6w\" (UniqueName: \"kubernetes.io/projected/560d5b6a-2afe-4d73-aede-b1d458609827-kube-api-access-8dx6w\") pod \"perf-node-gather-daemonset-9p99h\" (UID: \"560d5b6a-2afe-4d73-aede-b1d458609827\") " pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:30.705220 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.705141    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-pc2jz_da763fad-c5e4-4748-8c41-bc1c8c896ef3/volume-data-source-validator/0.log"
Sep 17 19:48:30.736949 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:30.736909    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:31.007557 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:31.007530    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"]
Sep 17 19:48:31.009991 ip-10-0-140-194 kubenswrapper[2587]: W0917 19:48:31.009961    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod560d5b6a_2afe_4d73_aede_b1d458609827.slice/crio-592a8292141cd5e68d8c2187baa6d1fed3e05dffeb45632e0b6a5a70e3e99f2c WatchSource:0}: Error finding container 592a8292141cd5e68d8c2187baa6d1fed3e05dffeb45632e0b6a5a70e3e99f2c: Status 404 returned error can't find the container with id 592a8292141cd5e68d8c2187baa6d1fed3e05dffeb45632e0b6a5a70e3e99f2c
Sep 17 19:48:31.470407 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:31.470318    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h" event={"ID":"560d5b6a-2afe-4d73-aede-b1d458609827","Type":"ContainerStarted","Data":"b9c0f17b2c1c2026f417d01fc652d196439aa410f06621ff681ad1e6af74ade8"}
Sep 17 19:48:31.470407 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:31.470369    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h" event={"ID":"560d5b6a-2afe-4d73-aede-b1d458609827","Type":"ContainerStarted","Data":"592a8292141cd5e68d8c2187baa6d1fed3e05dffeb45632e0b6a5a70e3e99f2c"}
Sep 17 19:48:31.470407 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:31.470400    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:31.487903 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:31.487840    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h" podStartSLOduration=1.487822973 podStartE2EDuration="1.487822973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:48:31.487320583 +0000 UTC m=+4163.366289352" watchObservedRunningTime="2026-09-17 19:48:31.487822973 +0000 UTC m=+4163.366791729"
Sep 17 19:48:31.753811 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:31.753791    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9r5ml_c65d126d-844f-4c22-b0c8-3747a48aa285/dns/0.log"
Sep 17 19:48:31.775788 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:31.775765    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9r5ml_c65d126d-844f-4c22-b0c8-3747a48aa285/kube-rbac-proxy/0.log"
Sep 17 19:48:31.912511 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:31.912487    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nngbm_10bad8ad-782d-4a52-8450-c3399c47dc9b/dns-node-resolver/0.log"
Sep 17 19:48:32.505735 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:32.505703    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-59srk_0010ab57-08d7-481d-aa7f-7c480d8ece9f/node-ca/0.log"
Sep 17 19:48:33.611010 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:33.610977    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7646b475dd-4rb55_ad67d410-cd73-46d3-81b8-d409d33ca41c/router/0.log"
Sep 17 19:48:34.168728 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:34.168693    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lv2sg_5378b81b-a0a2-4395-ab42-4faae5607cfc/serve-healthcheck-canary/0.log"
Sep 17 19:48:34.629367 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:34.629332    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-xdqjd_2af1f706-2aa2-4b4a-9960-1eb443e3d40e/insights-operator/0.log"
Sep 17 19:48:34.630018 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:34.630000    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-xdqjd_2af1f706-2aa2-4b4a-9960-1eb443e3d40e/insights-operator/1.log"
Sep 17 19:48:34.929525 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:34.929456    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ph9zh_857bafdf-f63a-4616-bb74-dfefd8f63fdf/kube-rbac-proxy/0.log"
Sep 17 19:48:34.955929 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:34.955906    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ph9zh_857bafdf-f63a-4616-bb74-dfefd8f63fdf/exporter/0.log"
Sep 17 19:48:34.977756 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:34.977735    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ph9zh_857bafdf-f63a-4616-bb74-dfefd8f63fdf/extractor/0.log"
Sep 17 19:48:37.485354 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:37.485324    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l7vd6/perf-node-gather-daemonset-9p99h"
Sep 17 19:48:38.047808 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:38.047778    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-54799454dd-khwgm_130a5513-c37f-4691-b6b8-74d817accee3/manager/0.log"
Sep 17 19:48:40.071397 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:40.071372    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cmwf4_5fc6ce9e-444f-4d6a-a586-5bab6eba20b4/s3-init/0.log"
Sep 17 19:48:40.089655 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:40.089631    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cmwf4_5fc6ce9e-444f-4d6a-a586-5bab6eba20b4/download-hf-model/0.log"
Sep 17 19:48:40.376652 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:40.376626    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7646b475dd-4rb55_ad67d410-cd73-46d3-81b8-d409d33ca41c/router/0.log"
Sep 17 19:48:41.562487 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:41.562447    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7646b475dd-4rb55_ad67d410-cd73-46d3-81b8-d409d33ca41c/router/0.log"
Sep 17 19:48:45.154709 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:45.154682    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-7p56c_987d9911-68e0-478a-a60d-234fbb2fbdc1/migrator/0.log"
Sep 17 19:48:45.176445 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:45.176411    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-7p56c_987d9911-68e0-478a-a60d-234fbb2fbdc1/graceful-termination/0.log"
Sep 17 19:48:45.526101 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:45.526067    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-dn9jp_3ba38230-b61f-4204-8b35-d60d7ee7b3b2/kube-storage-version-migrator-operator/1.log"
Sep 17 19:48:45.527423 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:45.527404    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-dn9jp_3ba38230-b61f-4204-8b35-d60d7ee7b3b2/kube-storage-version-migrator-operator/0.log"
Sep 17 19:48:46.760508 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:46.760474    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w4t6b_75b0ba53-5b75-493d-a60e-f479b53a0faa/kube-multus-additional-cni-plugins/0.log"
Sep 17 19:48:46.781370 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:46.781343    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w4t6b_75b0ba53-5b75-493d-a60e-f479b53a0faa/egress-router-binary-copy/0.log"
Sep 17 19:48:46.800875 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:46.800847    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w4t6b_75b0ba53-5b75-493d-a60e-f479b53a0faa/cni-plugins/0.log"
Sep 17 19:48:46.819962 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:46.819907    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w4t6b_75b0ba53-5b75-493d-a60e-f479b53a0faa/bond-cni-plugin/0.log"
Sep 17 19:48:46.840933 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:46.840898    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w4t6b_75b0ba53-5b75-493d-a60e-f479b53a0faa/routeoverride-cni/0.log"
Sep 17 19:48:46.863820 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:46.863792    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w4t6b_75b0ba53-5b75-493d-a60e-f479b53a0faa/whereabouts-cni-bincopy/0.log"
Sep 17 19:48:46.884323 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:46.884297    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w4t6b_75b0ba53-5b75-493d-a60e-f479b53a0faa/whereabouts-cni/0.log"
Sep 17 19:48:46.919993 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:46.919960    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fp6c4_05ce87e8-6de7-4fd2-bf76-ecef629fc3b2/kube-multus/0.log"
Sep 17 19:48:46.991003 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:46.990966    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7m9j8_b425f704-e370-447b-a2f4-23d81057111d/network-metrics-daemon/0.log"
Sep 17 19:48:47.015670 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:47.015578    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7m9j8_b425f704-e370-447b-a2f4-23d81057111d/kube-rbac-proxy/0.log"
Sep 17 19:48:48.454660 ip-10-0-140-194 kubenswrapper[2587]: I0917 19:48:48.454626    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pzlnd_9d5360ef-43f6-4b49-88bb-67b77b161636/ovn-controller/0.log"